Relationship Between HCAHPS Scores and Survey Response Rate Is Linked to Hospital Size.

Patient experience is an important dimension of health care quality and is assessed using the standard Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey for inpatients. The HCAHPS scores may vary based on survey response rate and hospital size. The objective of this study was to describe the association between survey response rate and HCAHPS scores and examine whether the relationship varies based on hospital size. Medicare's Hospital Compare publicly reported HCAHPS data were used. Pearson correlation, controlling for number of staffed beds, and linear regression models were used for the analysis. Hospitals were grouped into quartiles based on number of staffed beds to delineate the effect of increasing hospital size on the relationship between survey response rate and HCAHPS scores. A significant association between HCAHPS survey response rate and all examined HCAHPS domain scores was observed. The effect size across HCAHPS domains varied based on hospital size. The relationship between HCAHPS score and survey response rate differed significantly between hospitals in the smallest and largest size quartiles for discharge information, nurse communication, and hospital quietness. While a causal relationship cannot be inferred from this study, the response rate could be a direct and/or indirect driver of HCAHPS scores. Future research should be aimed to further explore the basis of this relationship and to determine how it may inform the interpretation of HCAHPS results.

Introduction

Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) represents the first national standardized publicly reported survey of patients’ perception of inpatient hospital care. The HCAHPS is a survey method of collecting and reporting perception of health care by patients to enable valid comparisons across all hospitals (1). With the shift from fee-for-service to increasingly value-based care models, Medicare reimbursement for inpatient hospital care is adjusted based on the quality of care, which includes an assessment of patient experience using this survey tool (2). Data collection of HCAHPS is mandated for hospitals that participate in Medicare Inpatient Prospective Payment Program, which includes value-based purchasing. Survey results are publicly reported on a quarterly basis.

Both survey response rate, an important determinant of the validity of results, and hospital size have been shown to have an impact on patient satisfaction ratings (3,4). Prior studies have shown that a higher survey response rate has been associated with higher patient satisfaction rating (3 –5); however, the patient satisfaction ratings were not based on HCAHPS (5). In the inpatient setting, small hospitals (defined by fewer than 100 beds) outperform larger hospitals (≥100 beds) with respect to HCAHPS scores (6). While scores are adjusted for patient mix and mode of survey data collection (eg, via telephone vs mail), scores at the hospital level are not adjusted for hospital size or survey response rate (7,8). The objective of this study was to describe the association between response rate and HCAHPS scores and examine whether the strength of the relationship varies based on the hospital size for inpatient care.

Methods

Medicare’s Hospital Compare online database of hospital-level HCAHPS results was used for this study, which was granted institutional review board exempt status. Data publicly reported on October 31, 2018, with records from January 1, 2017, to December 31, 2017, were included. HCAHPS domains examined were nurse communication, doctor communication, staff responsiveness, communication about medicine, discharge information, care transition, hospital cleanliness, hospital quietness, likelihood to recommend the hospital, and overall hospital rating. For each domain, reported “top box” scores were used and reflect the percentage of total submitted surveys for which the most positive answer was selected. For example, the “top box” response score is the percent of patients who reported that their doctor “Always” communicated well. This is the same for nurse communication, communication about medicine, staff responsiveness, hospital quietness, and hospital cleanliness (9). For overall hospital rating, the top box score was based on a score of 9 or 10 (on a scale of 1-10). For discharge information, the “top box” score is “Yes,” for likelihood to recommend the hospital, the “top box” score is “Definitely Yes,” and for care transition, the “top box” score is “Strongly Agree” (9).

The relationship between survey response rates and HCAHPS scores of hospitals was examined using a Pearson correlation. The data were then stratified into quartiles by hospital size based on number of staffed beds. Hospitals in the lowest quartile for hospital bed number were compared to hospitals in the highest quartile with respect to the relationship between survey response rates and HCAHPS scores. Linear regression models with a dummy variable were used to determine whether survey response rates predicting HCAHPS measures differed between the smallest and largest quartiles of hospital size as measured by bed number. The variables were standardized to a mean of 0 and standard deviation of 1. The analysis was repeated based on total annual discharges as a measure of size to corroborate the results. A 2017 data on number of staffed beds and annual discharges per hospital were obtained from American Hospital Directory (10).

Results

A total of 1707 hospitals were included and subdivided into quartiles based on number of staffed beds, resulting in 426 and 427 hospitals in the lowest and highest quartiles (“small” and “large” hospitals, respectively). The majority of the hospitals were nonprofit hospitals with emergency services provided. The smallest hospitals were majority critical care hospitals compared to acute care hospitals. For more information, please see Table 1 in the supplementary document. The mean (± SD) number of staffed beds for small hospitals was 22.72 ± 4.30 compared to 649.51 ± 302.58 for large hospitals. The mean survey response rate was greater in small hospitals compared to large hospitals (34.04% ± 10.66% vs 23.16% ± 6.77%, P < .001; Figure 1). Overall, there is a strong, significant, positive correlation between the HCAHPS variables and survey response rate, before and after controlling for hospital staffed bed numbers; the strongest being staff responsiveness (r = .600, P < .001). P values were all < .001 for each HCAHPS domain (Supplemental Table 2).

Table 1.

Linear Regression Model Associations Between HCAHPS Measures and Survey Response Rate Stratified by Hospital Staffed Bed Number.a

	Univariable Models Small Hospitals		Univariable Models Large Hospitals		Difference Between Regression Models?
HCAHPS Variable	Slope	95% CI	Slope	95% CI	Effect Size (t)	P
Discharge information	0.264	0.185 to 0.344	0.608	0.509 to 0.707	−5.055	<.001
Hospital quietness	0.346	0.266 to 0.426	0.022	−0.084 to 0.128	4.617	<.001
Nurse communication	0.304	0.227 to 0.381	0.449	0.354 to 0.543	−2.204	.028
Care transition	0.442	0.365 to 0.519	0.333	0.239 to 0.427	−1.663	.097
Overall hospital rating	0.368	0.294 to 0.441	0.449	0.344 to 0.553	−1.221	.223
Staff responsiveness	0.323	0.259 to 0.387	0.386	0.313 to 0.460	−1.184	.237
Hospital cleanliness	0.161	0.091 to 0.231	0.229	0.139 to 0.320	−1.132	.258
Likelihood to recommend hospital	0.405	0.326 to 0.483	0.457	0.351 to 0.563	−0.756	.450
Communication about medicine	0.284	0.203 to 0.365	0.269	0.187 to 0.351	0.228	.820
Doctor communication	0.238	0.155 to 0.321	0.233	0.148 to 0.397	0.083	.934

Abbreviation: HCAHPS, Hospital Consumer Assessment of Healthcare Providers and Systems.

a Small hospitals represent hospitals in the lowest quartile of hospital staffed bed number. Large hospitals represent hospitals in the highest quartile of hospital staffed bed number. The difference between the slopes of the regression models in large and small hospitals is reported. P values <.05 is considered statistically significant and are bolded.

Figure 1.

Box and whisker plot of survey response rate with respect to hospital staffed bed number. The bars represent the minimum and maximum survey response rate value, excluding outliers which are represented as dots. The X in the box represents the mean survey response rate, while the bar in the middle of the box represents the median survey response rate.

Similarly, based on hospital size, an effect size gradient across HCAHPS domains was noted. The strongest 3 correlations with response rate for small hospitals were care transition (r = .479, P < .001), likelihood to recommend the hospital (r = .440, P < .001), and staff responsiveness (r = .434, P < .001) while for large hospitals were discharge information (r = .506, P < .001), staff responsiveness (r = .447, P < .001), and nurse communication (r = .411, P < .001). Large hospitals did not have a significant correlation with hospital quietness (r = .02, P = .68; Supplemental Table 3).

In an analysis examining the impact of hospital size, based on number of staffed beds, the relationship between HCAHPS score and survey response rate differed significantly for discharge information, nurse communication, and hospital quietness between small and large hospitals (Table 1). A linear correlation between the remaining HCAHPS variables and survey response rate did not reveal a significant difference between small and large hospitals. To corroborate the results, the analysis was repeated using total annual discharges as a surrogate for hospital size, which revealed similar results (Supplementary Table 4).

To provide another measure of the strength of association, the slope, reflecting the incremental increase in score (rise or y-variable) with an associated increase in response rate (run or x-variable), was determined for each of the domains that differed significantly between small and large hospitals. Figure 2 shows the corresponding linear regression models for small versus large hospitals. A 10% increase in response rate was associated with a 6.1% increase in discharge information score for large hospitals versus only a 3.0% increase for small hospitals, yielding a slope that is 2.3 times greater than that for small hospitals. Similarly, for the nurse communication domain, the slope is 1.5 times greater for large compared with small hospitals. In contrast, for the hospital quietness domain, the slope is 15.7 times greater than that for large hospitals (although in the latter case there was no statistically significant relationship between HCAHPS score and survey response rate). The magnitude of these differences is illustrated graphically in bar charts in Figure 3, which shows the differences in mean change in score for discharge information, nurse communication, and hospital quietness per incremental change in survey response rate (ie, mean slope), categorized into quartiles by hospital staffed bed number.

Figure 2.

Standardized survey response rate versus (A) standardized discharge information, (B) standardized nurse communication, and (C) standardized hospital quietness stratified by hospital staffed bed number. Small hospitals are represented by hospitals in the lower quartile of hospital bed number and large hospitals are hospitals in the highest quartile of hospital bed number. *Indicates no statistically significant relationship exists between Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) score and survey response rate.

Figure 3.

Hospital staffed bed number separated into quartiles versus slope of the regression line representing an incremental increase in score (rise or y-variable) with an associated increase in response rate (run or x-variable) for (A) discharge information, (B) nurse communication, and (C) hospital quietness. Small hospitals are represented by hospitals in the lower quartile of hospital bed number and large hospitals are hospitals in the highest quartile of hospital bed number. **Indicates a P value of <.001. *Indicates a P value of <.05. Error bars represent standard error. *Indicates no statistically significant relationship exists between Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) score and survey response rate.

Limitations

There were several limitations. First, the presence of unaddressed confounders in this study could impact the results and should not be overlooked as sociodemographic and cultural factors could influence a patient’s likelihood to respond to the HCAHPS survey. Similarly, patient experience is another confounder that may influence both ratings and response, with satisfied patients more likely to fill out and return the survey. Additionally, a direct causal relationship between HCAHPS survey response rate and patient satisfaction scores cannot be inferred as this study is observational in design. Finally, patient’s perceptions of care may contribute to the hospital’s variation in HCAHP domain scores as it relates to survey response rate. Despite these limitations, HCAHPS is a standardized, national survey enabling the evaluation and comparison of hospitals with respect to patient experience, a vital component of health care quality, and hence it is important to understand trends linked to fundamental hospital characteristics.

Discussion

Previously published research demonstrated a positive association of HCAHPS survey response rate and patient satisfaction scores (11), and this relationship was observed in our study as well. Importantly, the strength of this relationship is linked to hospital size for some but not all domains. Larger hospitals had a significantly stronger relationship between survey response rate and discharge information scores as well as nurse communication scores. Smaller hospitals had a significantly stronger association between survey response rate and hospital quietness compared to large hospitals (which had no significant relationship). In an analysis using total annual discharges as an index hospital size, the same 3 HCAHPS domain slopes were significantly different between small and large hospitals.

While this cross-sectional study does not address whether HCAHPS survey response rate is an actual driver of a hospital’s scores, which remains a possibility. Maximizing survey response rate is a key objective in and of itself to ensure there is a valid representation of a hospital’s inpatient experience—not merely as an attempt to increase HCAHPS scores. Nonetheless, it is important to explore and develop a better understanding of these complex relationships impacting results of this standardized survey tool employed at all Medicare-participating hospitals. Such insight is of value to a variety of stakeholders, including individual hospitals committed to improving patient experience; data analytics, policy development, and value-based care oversight teams responsible for ensuring fair and effective methods for incentivizing the improvement of patient experience; and investigators studying the relationship between survey response rate and survey results in large data sets. Furthermore, given that driving survey response rate may be causally linked to higher scores for some domains, a deeper understanding may be necessary to understand whether the concordant increase in a hospital’s response rates and HCAHPS scores increase over time reflects actual improvement in patients’ perceptions of their inpatient experience.

If there were indeed a causal relationship between response rate and HCAHPS scores, response rate could be a direct and/or indirect driver of HCAHPS scores or could be linked to other factors broadly impacting both response rate and HCAHPS scores, such as organizational culture. Regardless of the explanation, it is possible that driving up survey response rate alone could drive up the average patient-reported hospital experience. While this is a testable hypothesis, clearly the overarching goal of HCAHPS is to improve the actual—not just apparent—patient experience of care at US hospitals. While the manner in which patients are engaged may be a confounding factor in efforts to drive up survey response rates and cannot be studied in the context of our study, quality improvement efforts moving forward could examine initiatives aimed at enhancing response rates and comparing approaches that would themselves inherently be expected to enhance experience of care. These approaches would, of course, need to be executed in a manner compliant with HCAHPS survey guidelines (12).

Another important implication of this work is that HCAHPS directed improvement efforts should be informed by the overall relationship between HCAHPS scores and the relative strength of the association for hospitals based on hospital size. Although survey response rate did not affect most of the HCAHPS domains when compared to small versus large hospitals, survey response rate does impact discharge information, nurse communication, and hospital cleanliness scores. Consequently, while it is important to consider the implications of this work with respect to potential improvement opportunities, such efforts should include goals of increasing HCAHPS scores and increasing actual patient experience.

Ideally, future work should further our understanding of the key factors underlying the relationship between survey response rate and HCAHPS scores. For example, one could implement highly formulaic, scripted reminders to patients to complete HCAHPS surveys with a message conveying a commitment to individual patient concerns, feelings, and responsiveness to patient needs (13 –16). This of course should include follow-up regarding specific concerns that are voiced. A study by Christensen et al found that sending reminders to cross-sectional, national health survey participants in Denmark can increase survey response rate by 30 or more percentage points (16). Because responsiveness of staff is the focus of one of the HCAHPS domains and because it correlates highly with scores in other domains (17,18), one might project that the intervention could positively impact staff responsiveness and perhaps even other scores. Additional interventions might be considered based not only on domains in which a hospital may be struggling but also on those for which, depending on hospital size, there is a stronger relationship to survey response rate.

Conclusions

Hospitals across the country are reimbursed for the quality of care provided, which includes patient experience as assessed by the HCAHPS survey tool. In an analysis determining the association between response rate and HCAHPS scores, it was found that HCAHPS variables had a positive, strong correlation with survey response rate. Specifically, large hospitals had a stronger, linear correlation between survey response rate and discharge information (2.3 times greater) along with nurse communication (1.5 times greater) compared to small hospitals. Small hospitals had a strong, linear correlation between survey response rate and hospital quietness, whereas there was no significant correlation for large hospitals.

Implications

There are 2 important implications of this work. One is that efforts to drive patient experience, whether at the organizational level or at the national level (such as through policy and financial incentives), should be informed by the relationship between survey response rate and HCAHPS scores and its propensity to vary with hospital size. Second, future studies aimed at developing a better understanding of these relationships could explore whether HCAHPS survey response rate an important driver of hospital scores. This would facilitate the development of strategies to improve actual—as opposed to apparent—quality of patient experience. In doing so, it will be important to account for the complex interplay among organizational culture, operational excellence, and other hospital characteristics that, in turn, may impact not only patient experience but also the overall quality and safety of patient care.

Click here for additional data file.

Supplemental Material, Supplemental_Doc for Relationship Between HCAHPS Scores and Survey Response Rate Is Linked to Hospital Size by Larissa G Rodriguez-Homs, Bradley G Hammill, Marc D Ryser, Harry R Phillips and Paul J Mosca in Journal of Patient Experience