Dedicated Assessors: description of an innovative education intervention to facilitate direct observation in the clinical setting.

BACKGROUND: The Department of Pediatrics at Queen's University undertook a pilot project in July 2017 to increase the frequency of direct observations (DO) its residents received without affecting the patient flow in a busy hospital-based pediatric ambulatory care clinic. Facilitating DO for authentic workplace-based assessments is essential for assessing resident's core competencies. The purpose of this study was to pilot an innovative education intervention to address the challenge of implementing DO in the clinical setting.
METHODS: The project allowed for staff physicians to act as "dedicated assessors" (DA), a faculty member who was scheduled to conduct direct observations of trainees' clinical skills, while not acting as the attending physician on duty. At the end of the project, focus group interviews were conducted with faculty and residents, and thematic analysis was completed.
RESULTS: Participants reported an increase in the overall quality of feedback received during the observations performed by a DA, with more specific feedback and a broader focus of assessment. There seemed to be little disruption to patient care. Some residents described the observations as anxiety-provoking.
CONCLUSIONS: Overall, this project provides insight into an educational approach that medical residency programs can apply to increase the frequency of workplace-based DO and boost the quality of feedback residents receive while maintaining the flow of already busy ambulatory care clinics.

Introduction

Direct observation (DO) is generally regarded as a key component of competency-based medical education (CBME) and could lead to more relevant assessment, more meaningful feedback provided by faculty, and improved overall resident competency.[1] Assessment following a direct observation is used far less than assessment based upon case review, [2] thus creating a gap between empirical evidence which indicates the importance of DO and actual practice in residency training.[2] There are numerous barriers to direct observation in the clinical setting including; lack of faculty development and comfort level of residents with direct observation, validity evidence for the assessment tools used, time required for observation, and remuneration for attending physicians.[2,3] Due to these barriers, substantial uncertainty remains on how best to practically implement or increase high-quality direct observation in the clinical setting in CBME residency training programs.[4]

Resource implications of the successful program transformations required for CBME are considerable (e.g., curriculum reform, assessment culture change, human resources and management, educational technology implementation). Although the assessment system is not the only major change, it does require considerable time and resources to operationalize. Being able to facilitate DO to allow for authentic workplace-based assessments is an essential part of implementing CBME as it is foundational to workplace-based assessment tools, allows for feedback provision as a powerful stimulus for learning, and strengthens the relationship and mutual trust between attendings and resident physicians.[4] This paper describes an educational innovation aimed at increasing DO by implementing a new role termed a Dedicated Assessor (DA) in an ambulatory clinic setting. It also explores assessor and resident perceptions of the project to inform best practices supporting DO as our residency training transitions fully to CBME.

Methods

Queen’s University transitioned all its (29) residency programs to CBME in 2017. We report here on the implementation and initial response to a “Dedicated Assessor” pilot as an innovative approach to increasing the quality of DO without impacting patient flow in a busy hospital-based ambulatory pediatric clinic. Ethics approval was approved via the HSREB, PAED-416-17 #6020971.

Dedicated Assessors (DA) are staff physicians who are scheduled to conduct direct observations of trainees’ clinical skills, but are not the attending physician on duty. In the pediatric ambulatory clinic, one attending physician typically runs an urgent care clinic with three to five assigned learners (medical students and residents). Attending physicians felt that taking time for DO and workplace-based assessments negatively affected patient flow through the clinic as it reduced the number of patients seen when staff physicians were also spending time directly observing residents in clinic. However, the high volume of patients with common general pediatric problems made the pediatric ambulatory clinic an ideal place to assess trainees.

Throughout the 18-month pilot project, DA scheduled one-hour blocks of time on a weekly basis to conduct DO of residents, while not working as the attending physician on duty. These DO were scheduled with a minimum goal of one resident assessed each day in the pediatric ambulatory clinic.

When completing any DO, our faculty used standard forms through our information technology platform to record and collate data to ensure there was increased consistency in the assessment of residents. The Department of Pediatrics has also worked on faculty development surrounding resident observations, shared mental model of assessment practices, and ensuring assessments are completed with appropriate narrative comments aimed at coaching residents for improvement. This process continued to be the practice with the new role of the DA.

Residents from two consecutive CBME cohorts were assessed; resident selection was mainly opportunity-based. This type of selection means that DA would attend the clinic at a particular time and assess whoever was the resident on duty. Residents were not always aware in advance that a DA observation was going to occur during their shift. Immediately after the observation, the resident and DA met to review feedback. Residents who participated received a DO assessment at least once during the project, but most were assessed in this manner 2-3 times over the course of the 18-month trial.

After the completion of the pilot project, focus groups were held with six of the seven faculty involved who acted as dedicated assessors over the 18 months. Residents (PGY1 & PGY2, five residents each year) from two consecutive CBME cohorts were invited to attend a focus group to provide their feedback and perceptions. Questions asked during the focus group were standardized and are found in Appendix A. Perceptions of the quality/quantity of feedback received, the overall perception of the DA role, and the impact on patient care/flow were all discussed during the focus group sessions.

The researchers transcribed the focus group interviews verbatim. Two of the researchers (AA and EH) coded the transcripts independently. The researchers followed the six-step process of qualitative analysis as outlined by Creswell to code data, generate themes and interpret the meaning of the results.[6] Both coders were engaged in the transition to CBME and used sensitizing concepts drawn from this experience. The authors discussed themes that emerged, and resolved any disagreements.

Results

For the purposes of this project, we defined an increase in quality to include 1) increased frequency of DO, 2) narrative feedback to the residents, specificity/personalization of those narrative comments, and 3) time spent discussing feedback with the resident at the time of observation (or soon after). Three main themes emerged from the analysis of focus group discussions: Increased quality of feedback, Faculty and resident perception of patient impact, and Perceptions of DA experience.

: Overall, residents, and faculty felt that this DA pilot project increased the quantity and the quality of the DO feedback they received. Residents explained their perception of quality increase by commenting that the feedback was, “definitely more personalized,” “it was more specific and included examples,” and “focused a lot more on your performance – which I think is really helpful.” When faculty were asked about their feedback to residents, they said that “the quality of my feedback is better from a DO versus a case review,” and “you can coach them on communication as well.” These observations support the premise that a dedicated assessor can improve the quality of feedback provided to residents by increasing narrative commentary, specificity, and personalization in feedback.

Most of the residents and faculty reported that they felt there was minimal impact on patient care or flow in the clinic. One of the DAs commented that, “as long as you are clearly defining what is happening in the room, patients have a good understanding that it is a teaching hospital.” Hence, patients were minimally distracted by the DO process.

It is important to note that at times, residents felt nervous about the presence of a DA in the room. One resident commented that “it’s kind of nerve wracking.” We attribute this may be due to the “opportunity-based” scheduling which meant that faculty scheduled their time, but just assessed whichever resident happened to be on duty. For example, one resident remarked, “if it is for my benefit, I am going to be much less anxious.” Faculty also picked up on the fact that this style of DO increased nervousness in the residents. However, over time “they get used to it and are less anxious” and the process becomes “viewed as coaching...not covert assessment of them.” Generally, residents reported that the increased quality of feedback received was beneficial and that with more exposure to the DA role, advance awareness of assessment opportunities and with clarity of purpose, they found the process less daunting than more formal DO assessment. Additional illustrative quotes from faculty and residents are included in Tables 1 and 2.

Table 1

Sampling of faculty responses to focus group questions

Faculty Perceptions
Changes to the quality and/or type of DO	“I think they got better feedback because before the pilot I wasn’t doing DO”“if a learner saw a patient and gave a case report, the type of feedback I could give before was on their organization or presentation. When you actually watch you can say “I noticed that….” And give more specific feedback”“I totally picked up things I wouldn’t from case review”“if you were able to observe the same resident you could build on the feedback you were giving”
Thoughts on resident perception	“they started off nervous; but after a couple of times they got used to it and were less nervous”“I wonder if the act of observing changes their behaviour? Maybe they are a bit more thorough”
Patient care experience	“I don’t think it changed much. No impact except to explain the reason”“I don’t think it affect flow of patients. Had to consider timing of when to give feedback after the DO so as not to interrupt flow”
General comments	“logistics and scheduling were a struggle”“having protected time scheduled is essential”“we are taking away the learner being the ones to initiate assessment”“being a dedicated assessor didn’t change the type of feedback I could give, but it made DO more like to happen”

Table 2

Sampling of resident responses to focus group questions

Resident Perceptions
Changes to the quality and/or type of DO	“bits of it are a little more staged; but I definitely liked when I had the same assessor more than once; they can see your improvements or give really constructive feedback”“some of the feedback has been more specific and included examples – compared to the general feedback you typically get at the end of clinic”“definitely more personalized feedback”
Thoughts on resident perception	“with DO the comment less on the ‘medical’ piece. DO helps them to evaluate (assess) can I develop rapport? Take a history?”“would be nice if we have objectives of what they were assessing for; no mutual understanding of what is being assessed”“more time spent as staff isn’t strapped for time”“it’s kind of nerve wracking – but I enjoy the feedback because otherwise you don’t really know how you are doing”
Patient care experience	“I wonder if it feels artificial for patients – do they feel like they are being watched? I don’t know if that changes their perspective?”“is there a perception that resident is not competent and therefore needs to be watched?”
General comments	“this has been really beneficial, as a 1st year I wonder am I doing it right; am I on the right track?”“we are really lucky to have this program”

Discussion

The need for DO in conjunction with robust feedback has been noted as one of the most critical components of CBME implementation.[6] Our DA innovation is a practical way to increase DO without adversely affecting patient flow due to the addition of faculty focused solely on assessment. DAs were introduced to help alleviate some of the stressors (time, comfort level, perception) surrounding workplace-based assessment in ambulatory clinic settings and thus far has proven successful. Understanding perceptions of DO can ensure its acceptance by stakeholders and successful implementation.[7] The key findings from this innovation can help medical educators to understand the perceptions of faculty and residents. Additionally, this pilot offers an approach to implementing DO in a successful way with the new role of a DA in busy clinical settings.

Barriers to DO include resources, institutional culture, faculty time/remuneration, inadequate faculty development related to assessment, and lack of validated assessment tools.[2,3,4] Barriers to implementation were minimized due to our program having already transitioned to a competency-based assessment consistent with the ‘Competence by Design’ framework.[8] At Queen’s University, our faculty do not work on a strictly fee-for-service model; therefore, the allocation of resources such as time and money minimally restricts our ability to schedule staff as dedicated assessors (DA) as much as has been reported in other research[3] in this area. Regarding other barriers, our department maintains several additional advantages: faculty have participated in multiple faculty development opportunities and receive ongoing training and information regarding assessment of residents and the Department of Pediatrics has in place standard assessment tools via an information technology platform that faculty are already using as part of resident assessment.

According to our findings, both faculty and residents spoke about increased specificity and quality of feedback as a result of the dedicated assessor innovation. Faculty discussed the advantage of DO as a tool to widen the scope of feedback provided, and residents echoed this perception. In terms of patient care experiences two different themes arose; residents did mention their concern that there was potential for DO to negatively impact patients’ perceptions of resident competence and faculty reported minimal impact on patient flow aside from the need to explain the purpose of DO to patients but also acknowledged the need to be strategic about when feedback delivery took place.

Residents did mention a need to negotiate the focus of assessment more explicitly which identified an opportunity for more education around ensuring residents are aware of the DA role. A recent study illustrated the performative aspects of direct observation for residents and suggested that direct observation is more valuable when faculty and residents have explicit expectations.[9] If assessment expectations are vague, residents could feel more nervous and potentially be less receptive to feedback from the DA. Participants identified logistics of scheduling faculty time for DA observations as a barrier for completing more DO from faculty and a situation we are still working on at Queen’s. Additional research to quantify concerns around DO and DA arrangement could be the next phase of this research.

Conclusion

The dedicated assessor approach is a practical way to increase the quality of DO residents received. Both the residents and faculty assessors reported greater satisfaction with feedback given/received and were appreciative of the increased time for DO and discussion. The process did not interfere with patient flow and allowed for feedback to be more learner centered and specific.

As the project continues increased faculty development is needed to ensure the focus of assessment is more explicitly negotiated between residents and faculty. Other barriers, such as logistics of scheduling, decreasing resident nervousness and ensuring there is ample time for discussion of feedback must also be addressed. Expanding this education intervention to other patient care areas, such as clinical teaching units and general ambulatory clinics may improve feedback to residents in those areas.