Difficult Interpersonal Encounters with Medical Students and Residents: Two Objective Standardized Teaching Encounters.

Introduction: Objective standardized teaching exercises (OSTEs) are widely used to develop professional competencies, especially in the health care professions. An OSTE involves exposing different providers to the same, time-limited scenario that is concurrently observed and/or recorded for either formative or summative evaluation. As there are limited resources available for creating a resident-specific OSTE, especially those applicable to family and community medicine residents, we created and evaluated a resident OSTE (R-OSTE) for second- and third-year family and community medicine residents.
Methods: This R-OSTE involved two cases. The first featured Taylor, a third-year medical student resistant to feedback. The second featured Kris, a first-year resident nervous about approaching the attending on duty. Our R-OSTE had residents teaching interpersonal skills to trained actors in a standardized learner role.
Results: Residents in the teaching role were formatively evaluated by peer observers (fellow residents) and standardized learners on interpersonal domains such as communication and professionalism. Learners gave residents an average performance rating of 4.9 on a 1 to 6 scale with 1 = Very Poor and 6 = Excellent. Residents also evaluated the OSTE itself, rating their experience on multiple teaching-related statements. Eighty-six percent of residents agreed this exercise was an appropriate development activity for family medicine residents. Overall, our R-OSTE was rated highly for relevance to teaching by the residents. Discussion: The residents were rated highly by both peer observers and standardized learners. However, there was little variability in peer observer scores, indicating the need for an alternative method of measurement.

Educational Objectives

By the end of both standardized teaching cases, residents will be able to:

Communicate with a resistant learner in an effective way.

Provide constructive feedback for a case presentation.

Use effective and positive body language during discussion.

Teach the learner how to effectively approach more senior doctors.

Address learner's lack of confidence when approaching more senior doctors.

Provide appropriate encouragement.

Devise a plan moving forward with the learner (e.g., senior resident plans to meet with first-year resident later in the day to follow-up).

Introduction

Interpersonal and communication skills comprise one of the six core competencies identified by the Accreditation Council for Graduate Medical Education (ACGME).[1] In order to be accredited, residency programs must demonstrate their curriculum integrates this competency. However, this competency can be challenging for residency programs to measure objectively.[2] Specifically, the competency states, “Residents must demonstrate interpersonal and communication skills that result in the effective exchange of information and collaboration with patients, their families, and health professionals.”[2] For years, our residency program used the objective standardized clinical exercise (OSCE) to formatively assess the patient and family portion of this competency and its related milestones (i.e., milestones C-1 and C-2).[3] However, only recently after conducting a faculty objective standardized teaching exercise (OSTE) did we see the utility of a standardized exercise to meet the health professionals' portion of this residency competency and its related milestone (i.e., milestone C-3).

The OSTE was introduced over 2 decades ago as a way to create a credible demonstration of the need for faculty to improve their teaching skills.[4] Since its inception, the OSTE has been adapted in myriad ways, including focusing on residents as teachers. Additionally, the OSTE has been identified as the gold standard for evaluating residents as teachers and training residents teaching skills.[5]

As with any standardized encounter, initially creating cases, training actors to perform in a standardized way, carrying out formative assessments, and figuring out logistics can be a daunting task. We looked to MedEdPORTAL to direct our case development. Specifically, Ricciotti's summary on implementing an OSTE with obstetrics-gynecology residents was helpful for outlining the logistics of an OSTE experience.[6]

While resources exist to create an OSTE, many of these focus on teaching clinical skills, and are not specific to family medicine residencies. This leaves the area ripe for study.[7-9] Further, as residents undoubtedly teach clinical skills to more junior residents and medical students they also often teach interpersonal skills such as how to interact with difficult patients, and appropriately approach an attending. There are currently no resources in MedEdPORTAL that focus on using the OSTEs with family medicine residents in order to assess and provide feedback on teaching interpersonal skills. Having OSTE scenarios specific to the residency type (e.g., family medicine) is recommended in the literature, beneficial according to previous research,[10] and, from our observations, creates a more realistic simulation for resident teachers.

Thus, we created the resident objective standardized teaching exercise (R-OSTE) to assess and improve the interpersonal and communication competency of family medicine residents. This was a formative experience, with the residents' OSTE performance intended to serve as a teaching exercise and not as an official resident evaluation. The target audience was second- and third-year residents in a family medicine residency program. Given the year of the residents and the goal of the experience, the OSTE was used to demonstrate performance on Level 3 and Level 4 of milestone C-3: “Develops relationships and effectively communicates with physicians, other health professionals, and health care teams.”[3] From our observations of recurrent problems over the years, we chose to focus on relationship development and effective communication with a learner displaying confidence problems. Specifically, we created two exercises that centered on learners with antithetical problems, one with too much confidence and another with not enough confidence. Residents participated in one and observed the other.

Methods

For the R-OSTE, standardized learners were recruited and trained to play Taylor, an arrogant third-year medical student, and Kris, a nervous and timid first-year resident. During the R-OSTE, family and community medicine residents participated in a standardized, 10-minute interaction with Taylor or Kris. Each interaction was unobtrusively observed via video by another family and community medicine resident who then completed a peer observer ratings of resident form. After the interaction, the observing resident led a 10-minute feedback session with the participating resident. After the feedback session, the standardized learner rated the resident on a standardized learner ratings of resident form. At the end of their participation and observation, all of the residents completed a resident evaluation of the OSTE experience. Lastly, a family and community medicine faculty facilitated a semi-structured debrief with all the residents to discuss and reflect on the experience.

Case A: Taylor

The Taylor presenting sheet (Appendix A) instructed residents that they would be listening to a case presentation from a condescending medical student. They were advised to focus on Taylor's oral presentation of the case and play their teaching role as residents. A suggested dialogue between Taylor and the resident is found under the teaching scenario. This guides the flow of the conversation and directs the role of the resident. Since the Taylor case focused on the content of a case presentation, the standardized learner training materials for the case included an extensive medical history of the patient. After the encounter, the peer observer guided the resident and standardized learner through a feedback session.

Case B: Kris

The Kris presenting sheet (Appendix B) instructed residents that they would be giving advice to a timid learner. The standardized learner was trained on a hypothetical dialogue between Kris and the resident. The teaching scenario illustrates the dialogue between learner and resident and contains suggestions on how the conversation should play out. After the encounter, the peer observer guided the resident and standardized learner through a feedback session.

Materials

Standardized learner ratings of resident

For the standardized learner's rating of the resident, standardized learners were given a checklist (Appendix C) to evaluate the role of the resident on multiple indicators, such as “appropriately explored my perspective” and “stated goals clearly and concisely.” Options on the checklist ranged from “very poor” to “excellent.” An area was included for open-ended comments at the end. This feedback is useful in determining the effectiveness of the OSTE along with potential applications of the OSTE in the future.

Peer observer ratings of resident

For the peer observer's rating of the resident, peer observers were given a behavioral checklist to mark whether or not a resident performed a variety of behaviors. Such behaviors included “uses eye contact” and “listened actively.” The six domains on our form included: (1) Establishes Rapport, (2) Maintained Relationship, (3) Learner Involvement, (4) Problem Solving, (5) Feedback, and (6) Supportive Communication. Each domain consisted of three or four specific, observable behaviors that the peer observer would check if the resident performed.

The format for our peer observer ratings of the resident is based in part on the domains of the Patient Centered Observation Form.[11] Domains pulled from this form included Establishes Rapport and Supportive Communication. Additional items on the peer observer ratings of resident were developed several years ago from the Common Ground Rating Form and our own personal experience,[12] with the domains of Maintained Relationship and Feedback being pulled form this form. Additional domains of Learner Involvement and Problem Solving were also included. Overall this is the first time the peer observer ratings of resident form has been used in an OSTE experience.

Resident evaluation of the OSTE experience

Residents were asked to rate their experience based on whether or not the OSTE was relevant to their teaching and/or practice. Responses were measured on a five-point Likert scale (1 = Strongly Disagree, 5 = Strongly Agree). An area for open-ended comments was also included (Appendix D).

Recruitment flyer

The flyer (Appendix E) was used to recruit standardized learners for the OSTE. It was created by the university-affiliated graphic artist.

Standardized learner training PowerPoint

The standardized learner training PowerPoint (Appendix F) was used for the first standardized learner training. It introduced the concept of the OSTE, provided structural and logistical details about the OSTE, and described the general roles the learners would be playing on the day of the OSTE. At the end of the PowerPoint, we showed videos from our previous faculty OSTE. These videos depicted both faculty-resident and faculty-medical student simulated interactions. Roughly, four to six videos were watched. Prior to the videos, the standardized learner educator went over the standardized learner ratings of resident evaluation with the standardized learners. While watching the video, the standardized learners completed the standardized learner ratings of resident evaluation. All evaluations are then reviewed as a group, which gave the standardized learner educator a chance to clarify any misconceptions about each domain on the evaluation. The video was not included in this resource to protect anonymity.

OSTE orientation for residents PowerPoint

The OSTE orientation for residents PowerPoint (Appendix G) gave residents an overview of the OSTE and specific details about the schedule for that day. The PowerPoint gave residents a tentative layout of how the OSTE would play out and outlined the type of feedback they would give and/or receive. Residents were taught to use the sandwich technique, which starts with positive feedback, goes into an area to improve, and end with another positive comment. A map of the clinical simulation laboratory was provided with details of where specific events would take place throughout the simulation.

Implementation

Schedule for OSTE

The schedule (Appendix H) included the time, date, and location of when the OSTE took place. It also included the schedules for the participants who are represented by numbers to maintain anonymity.

Facilitator debrief questions

Facilitator debrief questions were provided to faculty facilitators to serve as a guiding framework to lead the debrief session. Faculty facilitators added questions as they saw fit (Appendix I). The facilitator debrief questions were read by faculty facilitators prior to conducting the OSTE debrief, which occurred after all OSTE encounters and feedback sessions had taken place (Appendix I). The purpose of this debrief session was to help residents reflect on the experience and allow them to provide suggestions and criticisms for future OSTEs.

Standardized learner recruitment

A flyer was created for the recruitment of standardized learners (Appendix E). The flyer was distributed to a local community college's acting department as well as a regional state university's acting, nursing, and psychology departments. The flyer was also posted on the local university's job portal. Interested participants that contacted our department were interviewed. During these interviews, we inquired about past acting and medical experience. Ultimately, five nursing students from the regional state university were recruited as standardized learners.

Standardized learner training

Standardized learners were trained over two sessions (Appendix F). They were compensated $25.00 per hour for the two training sessions and their performance during the OSTE. The first training session consisted of: (1) a detailed orientation to the structure and logistics of the OSTE, (2) the assignment of the two cases, (3) a discussion of the cases, (4) tips and guidelines for being a standardized learner, and (5) watching two videos from previous OSTE events. The second training session consisted of acting out the cases both in the learner and teacher roles as well as watching three more videos from previous OSTE events.

OSTE event logistics

Prior to the OTSE, the research coordinator gave a 10-minute orientation to the resident learners (Appendix G). The orientation included an overview of the event, cases, and questionnaires to be completed. The resident OSTE took place in a simulation center that is commonly used by the residents to practice clinical skills. We used four of the exam rooms to act out the cases, an observation room for the peer observers, and an evaluation room for the residents to provide feedback on their experience in the OSTE (see Appendix G for a map). Each case was observed via video and headphones by another resident.

Each resident participated in one of the encounters as a teacher and one of the encounters as a peer observer. Thus, learners were exposed to two encounters. While in the peer observer role, the resident used an observation checklist to record behaviors performed by the resident teacher they were observing.

Each standardized learner was expected to participate in four encounters during the first OSTE day and three encounters during the second OSTE day. The number of encounters was based on the number of residents available to participate on the first day and the second day. After each encounter, the standardized learner filled out an evaluation of the residents' performance.

Each encounter consisted of a 10-minute interaction between the teacher and learner, a 10-minute feedback session led by the peer observer, and a 5-minute break to transition between encounters (Appendix H). Resident teachers were instructed when to enter the room, leave the room, start the feedback session, and end the feedback session via a speaker system connected to each simulation room. Additionally, 8 minutes into each encounter a 2-minute warning was given to let residents and standardized learners know the interaction was almost over. At 10 minutes, the residents were told the stop the interaction and leave the room.

After all encounters had been completed for the day the residents were taken to a separate classroom to participate in a debrief with a family medicine clinical faculty member. The debriefing was semi-structured with specific, open-ended questions. The faculty facilitator was able to add questions based on the direction of the conversation. See Appendix I for specific debriefing questions. The faculty facilitator wrote down brief notes based on resident comments. Additionally, these written comments were explored by the faculty facilitator in more detail during an in-person meeting with the OSTE coordinator. All comments were kept anonymous.

Schedule

For the first OSTE day, orientation went from 1:30 PM to 2:00 PM. Encounters began at 2:15 pm and ended at 3:50 PM, and the debriefing went from 4:00 PM to 4:40 PM. A schedule is provided in Appendix H. For the second OSTE day, orientation went from 1:30 PM to 2:00 PM, encounters began at 2:15 PM and ended at 3:25 PM, and the debriefing ran from 3:35 PM to 4:15 PM.

Staffing

Three staff members were required to facilitate the OSTE event. One simulation technician was required to monitor recording and fix technical difficulties. One individual was required to conduct the orientation and make sure residents arrived at the correct simulation room on time. An additional individual was required to make sure standardized learners were in the correct simulation room by 2:00 PM and to announce time milestones.

Results

Twelve residents in their second year and 14 residents in their third year of a family medicine residency program served as the teachers and peer observers. Standardized learners included five nursing students from the local state college.

There were a total of five standardized learners participating in this exercise. They were each asked to rate resident performance on seven indicators, on a global item, and on the hypothetical likelihood of approaching the resident in the future.

Standardized learners responded to all indicators and the global rating using a 6-point Likert-type scale (1 = Very Poor, 6 = Excellent). See Table 1 for a distribution of these ratings. On the global item, standardized learners gave residents an average performance rating of 4.9. The average score given to second-year residents was 4.7, while the average score given to third-year residents was 5.2 (p = .067). This indicates a trend toward better performance of residents who had been in the family medicine residency program longer. When asked the likelihood of approaching the resident in real life, learners indicated that this was “Likely” to “Very Likely” for the majority (69%) of residents (See Figure for breakdown). Separately, the peer observers were asked to rate the residents on six domains. See Table 2 for percentage of residents observed performing each behavior.

Table 1.

Five Standardized Learners' Ratings of Residents (N = 26) on Quality Indicators

Item	Very Poor	Poor	Fair	Good	Very Good	Excellent	Average Rating
Appropriately explored my perspective.	0.0%	0.0%	7.7%	23.1%	42.3%	26.9%	4.8
Effectively gathered information.	0.0%	0.0%	0.0%	26.9%	65.4%	7.7%	4.8
Expressed respect for me.	0.0%	0.0%	0.0%	19.2%	46.2%	34.6%	5.0
Listened actively.	0.0%	0.0%	3.8%	23.1%	42.3%	30.8%	4.8
Met my needs.	0.0%	0.0%	0.0%	38.5%	42.3%	19.2%	4.7
Provided positive and corrective feedback.	0.0%	0.0%	3.8%	26.9%	38.5%	30.8%	4.8
Stated goals clearly and concisely.	0.0%	0.0%	7.7%	26.9%	53.9%	11.5%	4.6
Overall rating.	0.0%	0.0%	0.0%	23.1%	53.8%	23.1%	4.9

Six-point Likert-type scale (1 = Very Poor, 6 = Excellent).

Figure.

Learner likelihood of approaching the resident in the future. Five standardized learners rated 24 residents.

Table 2.

Peer Observers' Indication That Residents (N = 23) Performed Key Behaviors

Domain	Behavior	Behavior Average	Domain Average
Established Rapport	Humor or small talk	33.3%	77.8%
	Uses eye contact	100%
	Warm greeting	100%
Maintained Relationship	Established learning environment	95.2%	96.8%
	Demonstrates presence; not rushed	95.2%
	Listened actively	100%
Learner Involvement	Engaged learner; avoided domination	94.7%	87.4%
	Encourage learner to learn on own	75.0%
	Explored learner perspective	95.0%
	Identified learner needs	85.0%
Problem-Solving	Effectively gathered information	95.0%	82.5%
	Summarized periodically	85.0%
	Open-ended and close-ended questions used	95.0%
	Stated goals clearly and concisely	55.0%
Feedback	Provided corrective feedback	84.2%	89.5%
	Provided positive feedback	94.7%
	Provided specific feedback	89.5%
Supportive Communication	Repeats/reflects	84.2%	84.2%
	Continuer phrases (e.g., Hmm, ummm, uh huh)	89.5%
	Verbal and nonverbal empathy	78.9%

There was no statistically consistent relationship between standardized learner and peer observer ratings. There were five domains on the two measures in which there was substantial content overlap: (1) Explored Perspective, (2) Effectively Gathered Information, (3) Listened Actively, (4) Met Needs, and (5) Stated Goals Clearly and Concisely. When these five domains were compared amongst raters, the reliability was poor for each domain. Intraclass correlation coefficients were all less than .5.

To evaluate the effectiveness of this OSTE, residents were asked to give suggestions based on their experience (Appendix D). See Table 3 for a distribution of resident ratings. All residents indicated they had experienced situations similar to the Taylor case or Kris case in real-life interactions. Forty-six percent of residents (12 out of 26) had experienced both cases.

Table 3.

Distribution of Resident Agreement on Evaluation of Experience

Item	n	Strongly Disagree	Disagree	Neutral	Agree	Strongly Agree
This exercise is an appropriate resident development activity for family medicine residents.	22	4.5%	0.0%	9.1%	63.6%	22.7%
This activity was relevant to my teaching.	22	0.0%	4.5%	13.6%	50.0%	31.8%
This activity has helped or will help me improve my teaching skills.	22	0.0%	4.5%	31.8%	40.9%	22.7%
This exercise helped me identify areas in which I would like additional training and/or education.	22	4.5%	0.0%	31.8%	45.5%	18.2%
This exercise helped me identify practices in my teaching that I would like to change.	21	0.0%	4.8%	42.9%	52.4%	0.0%

When asked to identify teaching practices that should be changed, many resident learners agreed that they would like to obtain more experience as a teacher. Some goals they set for themselves included offering more constructive feedback to people they disagree with, setting an agenda and specific expectations with their students, and using more of a holistic approach with the student. The resident learners were also asked to identify areas in which they would like additional training. For the most part, they requested more practice with teaching, more workshops on teaching methods, and more difficult situations with residents as well as other staff. Additionally, they were asked if there should be any changes in this OSTE in the future. Out of 17 resident learners, 15 said they would not make any changes or left the option blank. However, some suggested that instead of “teacher” and “learner,” use “intern” or “med student.”

Discussion

Difficult interpersonal encounters are relatively commonplace in the emotionally charged world of medicine and medical education, so every residency program is tasked with teaching its learners how to handle these. While these competencies are clearly delineated by the ACGME, how to model them or engage learners to develop them is often not as clear. The OSTE has been validated for developing clinical teaching skills in medical students, residents, and faculty.[13-17] These teaching encounters can often provide a real-world space for challenging residents in the realms of communication and professionalism, and the OSTE has been used effectively for well over a decade to assess and build professionalism in students.[18,19]

In the Fall of 2016, 26 family medicine residents participated in an R-OSTE. The R-OSTE involved two cases, Taylor and Kris. Given that our department has experience with standardized patient scenarios and faculty OSTEs, we began the design process 3 months in advance. This left ample time to recruit and train standardized learners as well as develop two new cases for the R-OSTE. To gather multiple perspectives, we used a three-prong method of evaluation: standardized learner ratings of residents, peer observer ratings of residents, and residents' own ratings of the experience. Furthermore, we conducted debrief sessions where residents were able to share their opinion on the experience with clinical faculty who had no involvement in the OSTE design and implementation. This provided qualitative information about pros and cons of the OSTE that was not captured on evaluation measures.

Overall, residents performed the majority of behaviors as observed by peers and were rated highly on interpersonal domains by standardized learners. By standardized learners, residents were rated highest on “expressed respect for me” and lowest on “stated goals clearly and concisely.” By peer observers, residents were rated highest on “maintained relationship” and lowest on “established rapport,” which was driven primarily by making small talk or using humor. We found that there was no correlation between standardized learner ratings and peer observer ratings, suggesting that there may be a ceiling effect on peer observer ratings or a response bias since the peer observers and residents often have close relationships due to time spent and challenges shared with one another.

When evaluating the experience, residents were the least likely to agree that “This exercise helped me identify practices in my teaching that I would like to change.” Roughly 43% responded as neutral, and 5% responded with disagree. In conversations with residents during the orientation and debriefings with faculty members, residents reported that they did not consider advising someone through an interpersonal issue as teaching. From previous OSTE research, we know that learners recognize and appreciate the need for direct observation and feedback, and that structured encounters can effectively be used as a teaching tool.[20,21] However, in contrast to our focus on interpersonal issues, these previous OSTE studies focused on more procedural education teaching. In light of these previous studies, the teaching terminology for our R-OSTE may be misleading and need to be changed. Alternatively, the emphasis on physicians as natural leaders and, therefore teachers, may not be getting reinforced or even mentioned in other clinical experiences. For other residency programs considering this resource, special attention needs to be paid to highlighting the importance of the interpersonal teaching that occurs or modifying the nomenclature.

Furthermore, several residents had experienced similar scenarios in real life, which confirmed the validity of our scenarios, especially the Taylor case. However, the experience still led to new, useful information that will inform curriculum development. Specifically, several residents expressed the need for more teaching experiences and teaching workshops to prepare for their future roles as educators. This information has been disseminated to program leadership as they consider changing our curriculum from block rotations to a longitudinal design.

Limitations

One limitation is our sample size. We only conducted one iteration of these two R-OSTE cases involving 26 residents. While two different family medicine residency programs were involved, we were limited in our ability to extrapolate these findings to a wider audience. As Cook et al. demonstrated in their systematic review of OSCEs, the use of only a single-group, single-assessment design is pervasive in simulation literature and creates poor methodological quality.[22] While we conducted a teaching simulation, rather than a clinical simulation, this quality issue remains.

Furthermore, there was a lack of a relationship between the resident performance rated by peer observer and the resident performance rated by standardized learner. There are a few suspected reasons as to why this may have occurred. One possibility is that the peer observers may have been influenced by their closeness with the individuals whom they were rating. In order to explore this possibility, we plan to have family medicine faculty serve as observers in a future OSTE. A second possibility is that the standardized learners may have not been adequately trained on evaluating the resident performance. While research demonstrates that standardized learners are as reliable as subject matter experts, this is based on the assumption that the standardized learners are well-trained.[13,23] For a future OSTE, we plan on developing a behavioral rubric corresponding to the domains on the resident performance as rated by standardized learner to use when training standardized learners.

Practically, the greatest challenge experienced was the human factor, both on the resident side and the standardized learner side. Specifically, two of the standardized learners did not show or give notice of their absence the day of the OSTE. This experience further supported our program's current practice of having an extra standardized learner present to substitute for any no-shows the day of the OSTE. Our OSTE had one coordinator and a support staff; the need for last minute reorganization of residents and standardized learners underscores the utility of additional support staff to rearrange the schedule as needed, and physically walk residents from the waiting area to the case rooms.

Future Directions

Given that we conducted this R-OSTE with second- and third-year residents, we will be able to reuse these scenarios starting with the 2018 R-OSTE. We plan on doing a second iteration in Fall 2018 that corrects limitations observed during this first iteration. Based on feedback received during the resident debrief, we are going to modify the Kris case. Residents expressed that this scenario was not challenging and did not understand why Kris would be nervous. Thus, we will paint the attending, Dr. Rodriguez, whom Kris is nervous to approach, as “known for being hard to approach” and “dismissive of resident concerns.”

Lastly, future iterations of this R-OSTE should consider utilizing actual residents as standardized learners. As Scott et al. addressed in their MedEdPORTAL OSTE resource, there is significant benefit to participating as a standardized learner in an OSTE.[9] This benefit should undoubtedly be given to the residents for whom the R-OSTE itself was created. Ultimately, we encourage others to use and modify this resource as they see fit. The most important future for this resource is that it is disseminated widely so that residency programs, especially family medicine residency programs, are aware that there is a guide for creating a satisfying OSTE experience for their residents.

A. Case A Taylor.docx

B. Case B Kris.docx

C. Standardized Learner Ratings of Resident.docx

D. Resident Evaluation of the OSTE Experience.docx

E. Recruitment Flyer.pdf

F. Standardized Learner Training.ppt

G. OSTE Orientation for Residents.ppt

H. Schedule for OSTE.xls

I. Facilitator Debrief Questions.docx

All appendices are peer reviewed as integral parts of the Original Publication.