Structured, Timely, and Individualized Patient Feedback to Residents and its Effect on HCAHPS Scores.

Physician communication is integral for improving patients' experience and clinical outcomes. Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) scores are used by the Centers for Medicare & Medicaid Services to assess the quality of care provided to the patients. We conducted a prospective pre-post-intervention study from March 2018 to March 2020 and included patients and residents of the community-based university-affiliated Internal Medicine Residency Program. Residents' communication was evaluated through resident surveys and results were shared with them and attending physicians within 48 h. To correlate with the effect over a 4-year period, 839 HCAHPS surveys were obtained for attending physicians as an outcome measure. The mean overall HCAHPS score and doctor's communication scores increased by 8.52 and 6.06 points post-intervention. The doctor's performance score for respect increased by 6.18 points, listening increased by 3.12 points, and explanation increased by 8.23 points. This study demonstrates that timely, structured, and individualized resident feedback in an academic medicine service can result in a sustainable increase in the attendings' overall HCAHPS and doctor's communication scores.

Introduction

Effective communication with patients is integral to patient-centered care (1). Physicians play a crucial role in communicating the plan of care with patients and impact the patient's overall care experience and clinical outcomes (2–4). Accreditation Council of Graduate Medical Education includes interpersonal and communication skills as one of the 6 core competencies for residents (5). Even though communication skills are enhanced in medical school, many residents are inadequately prepared for effective doctor–patient interactions (6).

Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) scores are used by the Centers for Medicare & Medicaid Services (CMS) to assess the quality of care provided to the inpatient Medicare beneficiaries and are tied to the value-based purchasing pay-for-performance metric. Ever since the first report was released on March 28, 2008, the HCAHPS scores are released by CMS on a quarterly basis (7). The HCAHPS survey evaluates communication with nurses, doctors, staff responsiveness, communication about medicines and discharge information, patients’ understanding of care, hospital environment, overall rating, and willingness to recommend others (8). The HCAHPS survey is administered to a random sample of adults that are 18 years or older, medical, surgical, or maternity care patients with nonpsychiatric primary discharge diagnosis, and inpatient admission with at least an overnight stay between 48 h and 6 weeks after discharge (7–9). In 2013, CMS began Hospital Value-Based Purchasing Program that withheld Medicare Severity Diagnosis-Related Groups Payments. These reductions have doubled from 1.0% in 2013 to 2.0% in 2017. The money withheld is redistributed based on the hospital's total performance scores (10,11).

HCAHPS scores are reported for attending physicians, even though residents provide most of the direct patient care in an academic setting. Only 1 study has evaluated the impact of residents’ feedback on attending's HCAHPS score. In 2011–2012, Banka et al. (12). conducted a pre–post-intervention study that involved residents’ education, assessing residents’ communication, and patients’ feedback to residents. In addition, performance incentives were given to high-performing residents (12). Our study was designed to expand and improve on the residents’ feedback process and was conducted over a 2-year period to assess the sustainability of the intervention in a residency program. The objective of this pre–post-intervention study was to assess the impact of structured, timely, and individualized patient feedback on the internal medicine (IM) attending physicians’ overall HCAHPS, doctors’ communication scores, and the HCAHPS subcomponents of respect, listening, and explanation over 2 years.

Methods

Study Design

In this prospective study, we evaluated the effect of providing residents with individualized education and feedback on the IM academic attendings’ overall HCAHPS and doctors’ communication scores which is a part of HCAHPS scores. We compared pre-intervention data from March 2016 to March 2018 with post-intervention data from March 2018 to March 2020. The study was approved by the Sparrow and Michigan State University (MSU) Institutional Review Boards. Due to the potential of an unintentional effect on CMS pay-for-performance HCAHPS program, a CMS exemption was also obtained to conduct the study. We included adult patients who were admitted to the Sparrow-MSU IM Residency Service from March 2018 to March 2020 for 2 midnights (evidenced by 2 fully documented care notes by the same resident), had decision-making capacity, and were able to provide written informed consent. We excluded patients with any of the following: altered mental status, critical sickness, worsening clinical status on the day of the survey, or refusal to give informed consent. A trained research assistant administered the survey to patients using an iPad device.

Settings and Participants

This study was conducted at the main Sparrow Hospital facility involving the Sparrow-MSU IM residency program in East Lansing, Michigan. Sparrow Hospital is a level 1 trauma center with 845 beds. The IM academic service is the second largest IM service in the hospital caring for a daily census of 64 patients. The teaching service has 4 rounding teams, and each team has a cap of 16 patients. Moreover, every team consists of a senior resident, 2 interns, and a supervising attending. Both IM residents and rotating non-IM residents from other subspecialties were included in the study. We evaluated a total of 66 residents with most participants being interns (64%) and belonging to the IM residency program (67%) under the supervision of 17 different attending physicians over a 2-year period. The primary intervention was focused on the patients’ feedback of the residents because of the nature and structure of the academic service. In our university-based inpatient residency program, the doctor–patient communication is primarily done by the residents, specifically, the interns while the attending physicians have a more supervisory role. Therefore the quality of communication reflected in attending physicians’ HCAHPS is the direct reflection of communication provided by the residents.

Intervention

The intervention comprised of 3 components: (i) Residents’ education, (ii) Feedback from patients shared within 48 h, and (iii) Acknowledgment and gift card incentive. For residents’ education, 2 educational sessions were held for the IM residents each year to educate them regarding the best practices to communicate with patients. Each session was 30 min conducted by an attending physician who demonstrated and rehearsed best practices with residents. Both IM and non-IM residents were provided with the best practices index card prior to starting every IM rotation (Figure 1). The index card was developed using the Rush University Medical Center's Best Practices checklist (13) as a reference and it also served as a reminder to use the mnemonics of “Name, Understand, Respect, Support” (NURS) (14) and “Setting, Perception, Knowledge, Emotions, Summary, Strategy” (SPIKES) (15). NURS is a standard tool used for patient-centered interviewing skills focusing on courtesy and respect; whereas, SPIKES assists in dealing with challenging conversations (14, 15).

Figure 1.

Best practices index card.

The CICARE survey used by Banka et al. (12) is an acronym that stands for “Connect with patients, Introduce yourself and role, Communicate, Ask and anticipate, Respond, Exit courteously.” Referencing this survey template, we developed a modified “resident survey” to obtain feedback from the patients (Figure 2). Each survey consisted of 11 questions with answers rated on the Likert scale (3 = always, 2 = usually, 1 = sometimes, and 0 = never). There were 2 open-ended questions at the end of the survey for qualitative information that was not scored; however, the comments were shared with the residents. A trained research assistant administered the resident survey to the patients in a structured and anonymous manner. Each resident's performance was assessed 4 to 6 times in a 4-week block; multiple times throughout the study. Resident survey results were electronically shared with the care team (the intern, senior resident, and attending) within 48 h. Reminders were sent to the team members to review surveys in a timely manner and for the supervising attends to provide individualized feedback to the residents. During the individualized feedback session, the attending physician discussed the feedback with the resident, reenforced best practices of communications, and highlighted areas for improvement. Every quarter, the residents were stratified based on the No. of surveys and the highest score. The intern with the best performance was recognized via department-wide acknowledgment email and given a $25 Starbucks gift card. Only 1 intern was eligible to receive 1 gift card throughout the year; the intervention data were collected from March 2018 to March 2020 and were analyzed quarterly to identify best-performing interns.

Figure 2.

Structured resident survey completed by patients.

Primary Outcome

Comparison of mean overall HCAHPS scores and doctors’ communication part of HCAHPS scores with its 3 components (respect, listening, and explanation) pre- and post-intervention.

Secondary Outcome

Comparison of mean resident survey scores between interns versus senior residents, IM versus non-IM residents, and performance on different questions of the resident survey.

Statistical Analysis

An interrupted time series regression analysis was used to evaluate the effect of the “resident surveys” on the overall HCAHPS ratings, doctor's communication performance along with its individual components of respect, listening, and explanation skills. Average HCAHPS scores from March 2016 to March 2018 were compared with post-intervention scores from March 2018 to March 2020. The interrupted time series regression analysis was modeled as Yt = β0 + β1Tt + β2Xt + β3XtTt + ɛt, where Yt was the outcome measured each year, Tt was the time since the start of the study, Xt was a dummy variable that represents the intervention, and XtTt was an interaction term. The analysis is conducted via “itsa” in the Stata program, where the main outcomes were β2 and the 95% confidence intervals (CIs) (16).

A secondary analysis was done using the resident surveys to compare the survey results between different residents and evaluate the performance on different questions. To calculate a percentage score, the Likert scale was converted to a numeric scale. A two-sample t-test was used to compare the average survey scores between interns and senior residents and IM versus non-IM residents. The Cochran–Armitage test was used to determine if the percentage of residents with an average score greater than or equal to 95% on the “resident survey” increased across 27 surveys which was the maximum No. of surveys per resident. In this test, the percentage of residents with an average score of 95% or higher in week i were defined as pi and evaluated the following hypotheses: H0: p1 = p2 = … = p27 versus H1: p1 < p2 < … < p27. The continuity-corrected asymptotic test statistics, or z-statistics, was used to detect if there was an increasing linear trend.

Results

There were 839 HCAHPS surveys done over the course of 2 years (215 in 2016, 270 in 2017, 199 in 2018, and 155 in 2019). The study stopped in March 2020 due to the Coronavirus pandemic. We compared the average score of the HCAHPS before (n = 485) and after (n = 354) intervention. We found the overall HCAHPS rating score increased by 8.52 points (95% CI −0.72 to 17.76, P = .08), doctor's communication score increased by 6.06 points (95% CI −4.33 to 16.45, P = .26), doctor's respect score increased by 6.18 points (95% CI −5.11 to 17.47, P= .29), doctor's listening score increased by 3.12 points (95% CI −7.29 to 13.53, P = .56), and doctor's explanation increased by 8.23 points (95% CI −3.08 to 19.54, P = .16) (Table 1). We analyzed collected data for each year (Table 2) and plotted the predicted trend line over the 4 years (Figures 3 to 7).

Table 1.

Average Change Between pre–Post-Intervention for Overall Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) and Doctor's Communication, Respect, Listening, and Explanation.

	Average point change	P value	95% confidence interval (CI)
Overall HCAPHS	+ 8.52	.08	−0.72 to 17.76
Doctor communication	+ 6.06	.26	−4.33 to 16.45
Doctor respect	+ 6.18	.29	−5.11 to 17.47
Doctor listening	+ 3.12	.56	−7.29 to 13.53
Doctor explanation	+ 8.23	.16	−3.08 to 19.54

Table 2.

Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) Performance of Attending Physician from 2016 to 2019.

Outcomes	Year	Mean	Median	Standard deviation
Doctors communication	2016	66.78	66.32	14.37
	2017	63.99	68.03	21.03
	2018	70.05	70.42	13.71
	2019	72.83	70.13	26.03
Doctors explain	2016	55.24	55.17	14.74
	2017	62.58	60.71	15.25
	2018	63.33	69.23	22.8
	2019	70.94	68.42	27.78
Doctors listen	2016	66.32	61.54	19.42
	2017	69.06	66.67	17.29
	2018	69.62	66.67	8.78
	2019	72.01	80	27.72
Doctors respect	2016	76.36	75	16.34
	2017	73.53	83.33	24.06
	2018	83.14	83.33	11.92
	2019	79.13	84.21	28.23
Overall rating	2016	62.56	63.82	21.03
	2017	62.86	65.86	15.3
	2018	64.06	66.17	5.59
	2019	78.4	74.9	19.61

Figure 3.

Overall Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) rating from 2016 to 2020 with predicted trend line.

Figure 7.

Doctor explanation rating from 2016 to 2020 with predicted trend line.

Doctor communication rating from 2016 to 2020 with predicted trend line.

Doctor respect rating from 2016 to 2020 with predicted trend line.

Doctor listening rating from 2016 to 2020 with predicted trend line.

In this study, we had 688 patients as participants for the “resident surveys” but 157 (23%) had to be excluded due to not meeting inclusion criteria. The final sample size was 531 “resident surveys” over 2 years; we evaluated a total of 66 residents with most of the participants being interns (n = 42, 64%) and belonging to the IM residency program (n = 44, 67%). The statistical analysis showed that the interns had a significantly higher mean score than the seniors (Intern 92.88 ± 6.49 vs Senior 84.64 ± 14.99, P = .016) (Table 3). Moreover, there was no difference between the scores of the IM residents and non-IM residents (IM 88.37 ± 12.35 vs non-IM 92.90 ± 7.16, P= .065) (Table 3). The Cochran–Armitage test demonstrated that the percentage of residents with an average score of 95% or higher on the resident survey was 64% for the first survey which significantly increased to 100% in survey 27. The z-statistics for the Cochran–Armitage test was −2.57 and P value was .0040 (Figure 8).

Table 3.

Analysis of Resident Surveys.

	Sample size (n)	Mean	Standard deviation	P value
Resident level		%
Junior residents	42	92.88	6.49	.0162
Senior residents	24	84.64	14.99
Resident type		%
IM residents	44	88.37	12.35	.0646
Non-IM residents	22	92.29	7.16

Abbreviation: IM, internal medicine.

Figure 8.

Cochran–Armitage test displaying the percentage of residents scoring 95% or more on the resident surveys.

Discussion

In this study, we demonstrated an increase in attending's overall HCAHPS scores, doctor's communication scores, and its components of respect, listening, and explanation. Our study provides an example of a simple intervention that could serve as a cost-effective means to improve patient experience in other residency programs. In addition to influencing HCAHPS, our study provided a valuable tool for residents to improve their communication skills and hence patient care which are both vital Accreditation Council for Graduate Medical Education’s core competencies (18). Our finding of an increase in the overall HCAHPS score is similar to the increase in HCAHPS score found in Banka et al. (12) study. Banka et al. (12) performed a prospective, non-randomized, pre–post-interventional study with a control cohort from other departments. They included a total of 3089 HCAHPS patient cases; 993 patients in the IM cohort and 2096 patients in the control cohort from all other adult specialties and found an increase in doctor communication scores with its components of respect, listening, and explanation over a year (12). In contrast, we conducted a study in the department of medicine without a control group; however, our study was conducted for a longer duration and the findings show sustainability of improved HCAHPS scores with the intervention.

Compared to the 20-item CICARE survey used by Banka et al. (12) to assess the resident's communication skills, we simplified the survey by having 15-item questions. We noticed an overall improvement in the resident survey scores for all 15 items during the study period corresponding to an improvement in the residents’ communication skills. The positive slope of the predicted trend line from 2017 to 2018 is due to the increase in the scores in 2018, which was after the intervention started. Interestingly the doctor's listening component of the HCAHPS score showed an increase of 3.12 points, whereas the improvement in the other components was more prominent. In our observation among the 3 HCAHPS components, “listening” is the most subjective and challenging component to change, which is depicted by a flattering rise in the trend line. We found that the interns had a significantly higher mean score on the resident survey compared to the senior residents. This is likely due to the fact that interns were the residents responsible for most of the direct patient care and spent more time at the bedside compared to senior residents. These findings further demonstrate the effectiveness of our intervention which benefited both the IM and non-IM rotating residents in a similar manner.

We provided incentives to interns with the best performance to recognize the effort of the team members. The focus was on the first-year interns because of twofold: first, we wanted the impact to continue forward as interns get promoted to second and third years and second reflection on the true state of our academic program where the interns are the primary inpatient care providers spending the most amount of time with the patients. We do not think the gift cards had a significant impact on the results due to the nominal amount ($25) and distributed quarterly to the only one-time eligible intern.

The only study with a similar intervention of providing individualized resident feedback and evaluating the attending physicians’ HCAHPS scores in the current literature is Banka et al. (12); however, there have been other studies that have evaluated communication skills in various academic settings. An observational study that conducted a focused group interview among 41 residents in general practice and surgical residents demonstrated that these residents preferred when their training program recognized and acknowledged communication skills as well as when appropriate feedback was available. Additionally, psychometric analysis of a modified American Board of Internal Medicine's Patient Assessment survey was conducted on 130 residents among 23 surgical and nonsurgical programs. The residents evaluated themselves on the modified survey and the analysis demonstrated that it was reliable for self-assessment of communication skills. This study proposed a future evaluation to see if administering the modified survey to both patients and residents following a clinical encounter to compare the patients’ evaluation and residents’ self-assessment of communication and professionalism skills.

Limitations

Our study has limitations, such as the ones that are inherent to the design of a single-site, non-randomized study. The study design did not collect demographic information to evaluate stratified outcomes. Our goal was not to look at demographics because the population in the catchment area of the hospital had not significantly changed over the past several years. Not including demographics served practical purposes of reducing the patient health information collection, increasing feasibility, and expediting IRB approval and CMS exemption. We were unable to obtain reliable group-level data for the attending physicians before 2016, which could have also increased the sample size. We did not examine the effect of the 2 educational sessions held per year because of the limited effect of didactics alone over a long period of time. Most training programs have educational sessions on communication skills, but evidence has shown that they by themselves do not have a significant impact. The study had a small sample size and further research is needed for generalizability. If we had the full score breakdown of each individual HCAHPS survey and a larger sample size, it is likely there would be a smaller variance in our results. Unfortunately, the intervention had to be stopped in early 2020 due to the COVID-19 pandemic; thus, we did not include the data from 2020 in the statistical analysis. Future large-scale studies are needed to further explore the impact of specific resident-targeted interventions to improve patients’ experience in an academic setting.

Conclusion

We found a positive and sustainable increase in the attending physician's communication scores and overall HCAHPS ratings for the IM academic service when real-time, structured, and frequent feedback was given to the residents. Our study provides a simple yet valuable resource to improve the patient experience as well as improve the communication skills of the residents in academic programs.