Characterizing the Relationship Between Hospital Google Star Ratings, Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) Scores, and Quality.

Google searches for hospitals typically yield a Google star rating (GSR). These ratings are an important source of information for consumers. The degree to which GSRs are associated with traditional quality measures has not been evaluated recently. We sought to characterize the relationship between a hospital's GSR, its Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) scores, and Centers for Medicare and Medicaid Services (CMS) quality measures. We found a moderate association between a hospital's GSR and its HCAHPS score. The relationship between a hospital's GSR and CMS quality measures was statistically significant, but the magnitude was quite low. Our findings suggest that consumers should not use GSRs as a hospital quality proxy.

Introduction

Hospital searches on Google often provide consumers with a hospital’s Google star rating (GSR). Google is the most widely-used online source of healthcare ratings and reviews (1). Given the saliency of the hospital GSR, it is possible that consumers are interpreting it as a surrogate for quality and using it to help decide where to seek medical care, though to our knowledge the extent of this trend has not been evaluated in the academic literature. We performed a descriptive study to characterize the relationship between a hospital’s GSR, its Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) scores, and quality measures.

Methods

We used Selenium WebDriver (https://www.selenium.dev/) to scrape (or extract) data (mean GSR and number of reviews) in August 2020. All URLs were manually checked to ensure data were scraped from the hospital's Google review webpage, rather than that of an associated physician or clinic. We excluded pediatric, rehabilitation, long-term care, and specialty hospitals. Data on readmissions (2015–2018), safety and mortality (2016–2019), and HCAHPS scores (2019) were obtained from the Centers for Medicare and Medicaid Services (CMS) Hospital Compare website (2). Data on hospital teaching status and bed size were obtained from the 2017 American Hospital Association Annual Survey.

We generated composite scores using the weighted mean of observed/expected medical readmissions (acute myocardial infarction (AMI), chronic obstructive pulmonary disease (COPD), heart failure, and pneumonia), surgical readmissions (coronary artery bypass graft (CABG), and total hip or knee replacement), medical quality (mean of quintiles of adjusted 30-day mortality rates for AMI, COPD, stroke, heart failure, and pneumonia), and surgical quality (mean of quintiles of adjusted 30-day CABG mortality, PSI-04 score (CMS post-surgical mortality measure), and PSI-90 (CMS perioperative adverse event measure)), and then divided hospitals into quintiles for each of these four metrics (with a higher quintile representing higher quality). We tested for an association between the number of reviews at a hospital and the GSR. Hospitals with 10 or fewer reviews were excluded.

We dichotomized hospitals at 3.5 stars and compared hospital characteristics, HCAHPS scores (overall stars and likelihood to recommend score), and quality outcomes by quintile. We chose 3.5 stars as the cutoff to dichotomize between the top 20% and bottom 80% of hospitals (in keeping with using quintiles for outcome measures). Univariate regressions of dichotomous GSR category on each quality metric (as a quintile) were performed. We also characterized the relationship between mean GSRs (as a continuous variable) and the continuous composite quality scores with scatter plots and correlation coefficients. Analyses were performed using Stata 16.1 (College Station, TX). Ethical approval is not applicable to this article. This article does not contain any studies with human or animal subjects. There are no human subjects in this article and informed consent is not applicable.

Results

There were 4,437 acute care hospitals in our sample, of which 985 had 10 or fewer GSRs and 270 had none. For a given review, Google stars ranged from 1 to 5 (discrete values), and the overall mean hospital GSR was 3.14 (standard deviation (SD) 0.66). The number of reviews ranged from 0 to 1,656 (mean 103, SD 148 for all hospitals and mean 131, SD 157 for hospitals with >10 reviews) and was right-skewed. The number of hospitals in the quality composite scores ranged from 2,411 hospitals for surgical readmissions to 3,943 hospitals for medical quality (with variation due to censoring by CMS). The correlation between the number of reviews and GSR at a given hospital was 0.063.

At the hospital level, high GSRs were strongly associated with higher HCAHPS overall star ratings and likelihood to recommend. High GSR hospitals had statistically significant though minimally superior surgical readmissions, medical quality, and surgical quality (Table 1). Specifically, the average hospital in the ≥3.5 Google star group relative to the <3.5 Google star group was in a 0.16 higher surgical readmissions quintile (P < .05), a 0.26 higher medical quality quintile (P < .001), and 0.35 higher surgical quality quintile (P < .001). It was also larger and was more likely to be a major teaching hospital. Scatterplots of mean GSRs and observed/expected readmissions and quality composite scores (high score = low quality) were constructed, and correlation coefficients ranged from −0.065 for medical readmissions to −0.13 for surgical quality (Figure 1).

Table 1.

Associations Between Quality Metrics and Google Star Ratings.

	<3.5 Google Stars (N = 2,713)	≥3.5 Google Stars (N = 739)	Difference (<3.5 Google Stars vs ≥3.5 Google Stars)
Hospital characteristics
Bed size, mean (SD)	187 (190)	210 (276)	23**
Major teaching (N, %)	129 (4.8%)	81 (11.0%)	0.062***
HCAHPS scores
Overall star rating, mean (SD)	3.07 (0.85)	3.56 (0.86)	0.49***
Likelihood to recommend, mean score (SD)	87.1 (4.3)	90.3 (3.6)	3.21***
Quality quintiles, a mean (SD)
Medical readmissions	3.01 (1.41)	2.99 (1.43)	−0.024
Surgical readmissions	2.98 (1.40)	3.14 (1.47)	0.16*
Medical quality	3.02 (1.42)	3.28 (1.40)	0.26***
Surgical quality	3.23 (1.41)	3.58 (1.39)	0.35***

Abbreviations: HCAHPS, Hospital Consumer Assessment of Healthcare Providers and Systems; SD, standard deviation.

Higher quintile is associated with better quality (lower mortality/morbidity/adverse events, lower readmissions).

*Significant to P < .05.

**Significant to P < .01.

***Significant to P < .001.

Figure 1.

Relationship between hospital mean google star ratings and quality

Discussion

The correlation between a hospital's GSR and its HCAHPS scores was positive, though weaker than we had anticipated. Previous work evaluating the relationship between hospital ratings on various online review platforms and HCAHPS scores has shown a significant positive association (3–5). We observed a weak association between a hospital's number of reviews and its GSR.

The relationship between online ratings and quality has been less well-studied. While the positive correlation between relatively higher GSR hospitals and higher quality was statistically significant for three of the four composite metrics, the effect size was quite small. Our findings expand on prior work on GSRs (6) and Yelp reviews (5). Not only has the mean number of reviews per hospital doubled since the previous study (6), but this prior study evaluated only all-cause 30-day readmissions, 30-day mortality from pneumonia, and rate of colonic infections, while we focused on broader medical and surgical quality outcomes. A recent study of Facebook, Google, and Yelp hospital ratings and individual surgical outcomes (mortality, a composite score, and a number of specific perioperative complications) looking exclusively at large teaching hospitals found either non-significant associations or associations that were significant but of a very low magnitude (with higher ratings associated with lower rates of mortality and complications) (4). Our study differed in that it included all acute care hospitals and evaluated medical quality measures in addition to surgical ones.

Previous research on the relationship between HCAHPS scores and quality suggests a complex picture, though the majority of the literature suggests a positive association between patient satisfaction and high-quality care. One study evaluating the relationship between HCAHPS scores, Hospital Quality Alliance scores, and patient safety indicators found a significant and robust association between high HCAHPS scores and high-quality care (7). A narrative review (8) and systematic review (9) both observed an overall positive correlation between patient satisfaction and quality. However, another study showed an association between higher HCAHPS scores and increased mortality (10).

Our analysis suggests that GSRs are a poor predictor of CMS-defined quality. We utilized multiple publicly reported CMS measures to create each of our four quality composite scores. Of course, readmission, mortality, and adverse perioperative events are not the only measures of quality, though they are especially relevant because CMS publicly reports them and poor performance can trigger financial penalties. New and innovative ways for consumers to determine the quality of care at a hospital, especially ones that are updated in real-time, might be beneficial. Future studies could evaluate the relationship between GSRs and other measures of quality (e.g., rates of medication errors and misdiagnoses) and other non-quality measures such as healthcare costs.

Limitations

An important limitation is our inability to ensure appropriate attribution of these GSRs. We were not able to confirm that the reviewers actually received care at the hospital for which they submitted a GSR. Additionally, we cannot determine the type of care they received (e.g., care in the Emergency Department, a medicine service, or a surgical service).