"I Don't Trust It": Use of a Routine OSCE to Identify Core Communication Skills Required for Counseling a Vaccine-Hesitant Patient.

BACKGROUND: Vaccine hesitancy is challenging for clinicians and of increasing concern since COVID-19 vaccination rollout began. Standardized patients (SPs) provide an ideal method for assessing resident physicians' current skills, providing opportunity to practice and gain immediate feedback, while also informing evaluation of curriculum and training. As such, we designed and implemented an OSCE station where residents were tasked with engaging and educating a vaccine-hesitant patient. AIM: Describe residents' vaccine counseling practices, core communication and interpersonal skills, and effectiveness in meeting the objectives of the case. Explore how effectiveness in overcoming vaccine hesitancy may be associated with communication and interpersonal skills in order to inform educational efforts.
SETTING: Annual OSCE at a simulation center. PARTICIPANTS: 106 internal medicine residents (51% PGY1, 49% PGY2). PROGRAM DESCRIPTION: Residents participated in an annual residency-wide, multi-station OSCE, one of which included a Black, middle-aged, vaccine-hesitant male presenting for a routine video visit. Residents had 10 min to complete the encounter, during which they sought to educate, explore concerns, and make a recommendation. After each encounter, faculty gave residents feedback on their counseling skills and reviewed best practices for effective communication on the topic. SPs completed a behaviorally anchored checklist (30 items across 7 clinical skill domains and 2 measures of trust in the vaccine's safety and resident) which will inform future curriculum. PROGRAM EVALUATION: Fifty-five percent (SD: 43%) of the residents performed well on the vaccine-specific education domain. PGY2 residents scored significantly higher on two of the seven domains compared to PGY1s (patient education/counseling-PGY1: 35% (SD: 36%) vs. PGY2: 52% (SD: 41%), p = 0.044 and activation-PGY1: 37% (SD: 45%) vs. PGY2: 59% (SD: 46%), p = 0.016). In regression analyses, education/counseling and vaccine-specific communication skills were strongly, positively associated with trust in the resident and in the vaccine's safety. A review of qualitative data from the SPs' perspective suggested that low performers did not use patient-centered communication skills. DISCUSSION: This needs assessment suggests that many residents needed in-the-moment feedback, additional education, and vaccine-specific communication practice. Our program plans to reinforce evidence-based practices physicians can implement for vaccine hesitancy through ongoing curriculum, practice, and feedback. This type of needs assessment is replicable at other institutions and can be used, as we have, to ultimately shed light on next steps for programmatic improvement.

INTRODUCTION

Vaccine hesitancy is a challenging phenomenon for clinicians to tackle and of increasing concern since the COVID-19 vaccination rollout began. Studies have shown that despite underrepresented minorities having been disproportionately affected by COVID, they have a higher pooled prevalence of vaccine hesitancy compared with the overall adult US population.[1] Reasons for hesitancy are numerous, but include distrust of the healthcare system stemming from past discrimination (personal or historical) as well as historical disparities in access to information and health literacy.[2]

Given that we now have safe and effective vaccines to fight the COVID-19 pandemic, we felt it was essential to assess the skills of our medicine resident physicians in overcoming vaccine hesitancy. Although broad communication skills have been a core focal point of residency curricula and assessment, resident physician training on vaccine-specific communication has been lacking in training programs, with one study reporting that nearly a quarter of residents report never having learned these skills during training.[3]

Simulation, namely the use of standardized patients (SPs), provides an ideal method for assessing residents’ current skills, giving them an opportunity to practice; providing immediate feedback; and informing program-level evaluation of the residents’ curriculum and training experiences—all within a highly relevant, “just-in-time” framework.[4-7] As such, we designed and implemented an OSCE station where residents were tasked with engaging and educating a vaccine-hesitant patient. We used this OSCE case as a “just-in-time” assessment of our residents’ skills in overcoming vaccine hesitancy, provided immediate feedback to residents, and used the results to plan a future residency training. In this paper, we describe residents’ vaccine counseling practices, their core communication and interpersonal skills, and their effectiveness in meeting the objectives of the case. We then explore how effectiveness in overcoming vaccine hesitancy may be associated with core communication and interpersonal skills in order to inform education and training efforts.

SETTING, PARTICIPANTS, AND PROGRAM DESCRIPTION

Internal medicine residents (N = 106; 51% PGY1, 49% PGY2) participated in a series of routine OSCE encounters with SPs during the spring of 2021, one of which included a Black, middle-aged, vaccine-hesitant male presenting for a routine video visit (Table 1) (Appendix 1). Prior to this assessment, residents had approximately 6 months of experience conducting video visits. The challenge for the residents was to explore this patient’s underlying concerns, educate him about COVID and the vaccine, and help him make an informed decision. Residents had 10 min to complete the encounter. After each encounter, faculty had 5 min to provide residents with feedback on their counseling skills and review best practices for effective communication on the topic. All residents attended a group debriefing at the end of the OSCE, during which they discussed how best to manage hesitant patients and shared lessons learned from participation.

Table 1

Case Description

Case information
Patient age, race, gender	56, Black, male
Chief complaint	Requesting new prescription lenses
Medical history	• Former heavy smoker • High cholesterol • Family history of cancer • Up to date on routine vaccines • No allergies
Social history	• Works in nursing home • Lives with wife of 25 years • Father of four grown children
COVID-19 experience and vaccine reluctance	• Had a difficult year with COVID, family member almost died • Works in nursing home, has seen how bad it can be • Willing to discuss vaccine if prompted, but not interested because: ○ Believes it to be dangerous ○ Concerned about its quick development ○ Has heard of Tuskegee experiment ○ Does not trust the government’s involvement
Challenge to resident	• Discuss the COVID vaccine—indications, risks, and benefits • Negotiate a plan with patient

Prior to the OSCE, two SPs received a minimum of 4 h of case training in order to ensure accurate portrayal of character and assessment. Approximately 2 h was dedicated to character practice while the remaining 2 h was dedicated to checklist training and standardization of the scoring procedure.[8] Case training included a review of specific responses to potential conversational prompts regarding vaccines and concerns over safety.[9] After each resident encounter, SPs assessed resident performance using a checklist consisting of behaviorally anchored items (scale: not, partly, or well done) across six core clinical skill domains (Table 2) (Appendix 2). Included in these domains were three communication domains (information gathering, relationship development, education and counseling), skills in demonstrating empathy (recognizing, responding to, validating emotions), activating the patient (based on Hibbard’s[10] model of patient activation in which activated patients—those that understand their condition—are confident they can manage their health, and feel they can partner with their physician), and patient centeredness, and then case-specific vaccine education practices. Vaccine-specific assessment items were developed through a combination of our clinical leadership’s consensus and the work of Dudley et al.[11] on vaccine communication.

Table 2

Performance in Key Domains (N = 106 residents)

	Items	% of residents (N = 106) rated “well done”	Summary scoreMean (SD)	Cronbach’s alpha
Vaccine specific
Vaccine education and counseling practices	Introduced topic of vaccine appropriately	58%	55% (43%)	0.90
	Explored/elicited your underlying reasons for being hesitant about the vaccine	55%
	Addressed specific concerns	51%
	Discussed disease risk and transmission	51%
	Educated you about the vaccine	63%
	Gave a strong and personalized recommendation	50%
Core clinical skill domains
Communication: information gathering (4 items)	Elicited your story using appropriate questions	64%	64% (41%)	0.75
	Managed the narrative flow of your story	61%
	Clarified information by repeating to make sure he/she understood you on an ongoing basis	57%
	Allowed you to talk without interrupting	72%
Communication: relationship development (5 items)	Communicated concern or intention to help	58%	61% (44%)	0.88
	Non-verbal behavior enriched communication (e.g., eye contact, posture)	56%
	Acknowledged emotions/feelings appropriately	59%
	Was accepting/non-judgmental	66%
	Used words you understood and/or explained jargon	67%
Communication: education and counseling (3 items)	Asked questions to see what you understood	42%	44% (40%)	0.64
	Provided clear explanations/information	52%
	Collaborated with you to identify and decide on possible next steps/plan	35%
Empathy: emotions (3 items)	Encourages and is receptive to your expression of emotion	59%	59% (48%)	0.93
	Identifies feelings—makes clear attempts to explore your feelings by identifying/labeling them	56%
	Accepts feelings—clearly indicates acceptance/validation of your feelings	62%
Patient centeredness/satisfaction (4 items)	Was sensitive/responsive to my needs/situation	54%	55% (46%)	0.90
	Explored my expectations about visit (problem, solution)	60%
	Took a personal interest in me; treated me as a person	37%
	Made you feel like there was enough time (visit was not rushed)	48%
Patient activation (3 items)	This encounter helped me fully understand the nature/causes of my problem/health condition	51%	47% (47%)	0.91
	This encounter made you feel that you would be able to take control over your health	45%
	This encounter made me feel like an equal partner	47%
Case outcomes	% of residents (N = 106)
SP report that character trusted physician	62%
SP report that character would believe the vaccine was safe	62%

The percent of residents rated as performing “well” on each individual item is provided, and summary scores for each of the six skill domains and the specific vaccine-hesitancy practices domain were calculated as percent of items in a given domain rated by the SP as “well done.” Mean “percent well done” scores are provided for the full sample and by PGY. Two outcome measures assessed residents’ effectiveness in overcoming the patient’s hesitancy: (1) whether or not the SP (in character) trusted the resident physician and (2) if the SP (in character) believed the vaccine to be safe at the completion of the visit. Analyses included descriptive analyses, t tests for domains and differences based on post-graduate year (PGY), and regression to explore relationships between clinical skill domains, vaccine hesitancy practices, and case outcomes. For regressions, predictors were scaled as 0–1 and collinearity between variables was assessed prior to analyses. This study qualified as a quality improvement project based on NYU’s Institutional Review Board criteria.

EVALUATION

Individual Item and Domain Scores

In terms of individual vaccine education items, more than half of the residents performed “well” with 58% fully discussing vaccines, 55% exploring and 51% addressing the patient’s underlying concerns, 63% educating about the vaccine, and 50% giving a personalized recommendation (Table 2). On average, this sample of residents had a mean score of 55% well done in this domain but with substantial variation in individual learner performance (range: 0–100%; SD = 43%).

Cronbach’s alpha values for core domains ranged from 0.74 to 0.96, signifying high internal consistency. These values did not differ substantially between the two SP raters. Overall, residents performed better in the communication sub-domains of information gathering and relationship development (means > 60%) than they did in the communication sub-domain of educating the patient, and in the empathy, patient activation, and patient satisfaction domains (means < 60%) (Table 2). The lowest score was for the sub-domain of patient education and counseling: mean = 44% (SD: 40%). PGY2 residents performed significantly better in the communication domain of education and counseling (PGY2 mean = 52% (SD: 41%) vs PGY1 mean = 35% (SD: 36%), p = 0.044) and on patient activation (PGY 2 mean = 59% (SD: 46%) vs PGY 1 mean = 37% (SD: 45%), p = 0.016).

Commentary collected from SPs for the poor performers found that SPs noted that these residents tended to be a bit “pushy” about the vaccine, with one SP noting, “[They] did all of the talking. I felt like [they] gave me a science lesson.” Another stated that, “At times I felt judgement for my reasons for abstaining from the vaccine.” Further, SPs reported authoritative approaches to communicating vaccination importance as a negative, by stating that “‘I am your doctor,’ [is] not a great comment for encouraging the patient to get the vaccine” and “I felt that [they] talked AT me, not TO me. [They were] not personal in [their] speaking.”

Performance on Case Outcomes

Regarding case outcomes, 62% of SPs reported both trusting their resident physician at the completion of the visit and feeling safe about the vaccine upon completion of the encounter, with more PGY2 residents achieving both of these outcomes (64%) than PGY1 residents (48%) (p = 0.114). Clinical skill and COVID vaccine hesitancy practice scores were significantly higher for residents who were rated as trustworthy (compared to those who were not rated as trustworthy) in all instances (% difference ranged from 49 to 59% higher, p < .001 for all scores) and for those who convinced the SP that the vaccine was safe (vs those that did not convince the SP) (% difference ranged from 50 to 59%, p < 0.001 for all scores).

Regression Analyses

In logistic regressions with core skill domain scores as “predictors” and case outcomes as binary variables, we found that the communication domain of education and counseling skills was very strongly associated with SPs reporting that they trusted the resident (odds ratio = 35.2, 95% CI = 2.30–537.29, p = 0.010) and that they perceived the vaccine as safe (odds ratio = 174.3, 95% CI = 6.75–4499.66, p = 0.002). Scores on case-specific vaccine education were also important for establishing trust in the resident (odds ratio = 36.7, 95% CI = 1.20–1119.77, p = 0.039) and perceiving the vaccine as safe (odds ratio = 90.3, 95% CI = 1.93–4210.18, p = 0.022).

DISCUSSION

Results of this needs assessment suggest that while a little over half of the residents had strong skills in tackling vaccine hesitancy, many residents did not and required in-the-moment feedback. These same learners may benefit from additional education, training, and practice. While this OSCE case was developed to address the immediate need to increase COVID vaccination rates, addressing vaccine hesitancy more generally is not a new issue and is an essential competency for internal medicine physicians. Our learners have had limited formal training surrounding vaccine-specific communication (with the exception of ad hoc communications with preceptors). As a result of our findings, our residency program plans to reinforce the specific evidence-based practices physicians can implement to tackle vaccine hesitancy through ongoing curriculum, practice, and feedback (e.g., we plan to repeat this OSCE case in the upcoming year).

Beyond these specific practices, general communication skills in educating and counseling (explaining things clearly, checking understanding, and collaborating in next steps) appear particularly important to establishing trust and enhancing perceptions of vaccine safety. In addition, skills in activating patients (as adapted from Hibbard’s Patient Activation Measure[9]) are also very strongly associated with overcoming vaccine hesitancy. Residents’ core communication and interpersonal skills matter. Our finding that PGY2 residents performed better in these (and most assessed skill domains) when compared to PGY1 residents provides evidence that our program and residents’ experiences are helping them become more competent in these foundational areas over time.

In order to better inform further training, we reviewed the SPs’ responses to open-ended questions about the performance of the residents, focusing on those residents that did not achieve the case outcomes. Overall, the comments reinforced that these residents were not effective because they were pushy about the vaccination and did not use best practices[9] (exploring/eliciting reasons for hesitancy, addressing specific concerns), did not integrate foundational communication skills (focusing on explaining things clearly, asking what the patient understood, and collaborating in next steps), and did not focus on activating the patient (working to ensure patients felt confident in taking action) during the encounter.

This study is not without limitations. As these data were collected as part of a simulation, findings cannot be extrapolated to independent practice without clinical observation. Considering that our OSCE is a one-point-in-time assessment with a small sample of learners, longitudinal follow-up of skills in subsequent OSCEs with larger and more heterogenous samples is necessary for drawing stronger conclusions regarding skills required for vaccine counseling. Further, given our small sample size, data distribution, and mild collinearity between variables, the large effect sizes we found for the association between patient education and counseling and vaccine hesitancy practices and overcoming the patient’s vaccine hesitancy may be over-estimated. Nonetheless, this type of needs assessment and analysis can be replicated at other institutions and can be used, as we have, to provide insight into whether residents have the core skills necessary for engaging in communication and trust building with vaccine-hesitant patients, ultimately shedding light on next steps for programmatic improvement. In our case, these include providing specific vaccine counseling training for our learners, engaging in ongoing assessment of their mastery of skills during subsequent OSCEs, and continuing to emphasize the critical importance of educating and activating patients in achieving positive patient outcomes both in the context of simulation and real clinical practice.

(DOCX 23 kb)