Translation, adaptation and validation of the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) for use in Japan: a multicentre cross-sectional study.

OBJECTIVES: The Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) is a well-established and internationally recognised scale for measuring patients' experience with hospital inpatient care. This study aimed to develop a Japanese version of the HCAHPS and to examine its structural validity, criterion-related validity and internal consistency reliability.
DESIGN: Multicentere cross-sectional study.
SETTING: A total of 48 hospitals in Japan. PARTICIPANTS: 6522 patients aged ≥16 years who were discharged from the participating hospitals.
RESULTS: Confirmatory factor analysis showed excellent goodness of fit of the same factor structure as that of the original HCAHPS, with the following composites: communication with nurses, communication with doctors, responsiveness of hospital staff, hospital environment, communication about medicines and discharge information. All hospital-level Pearson correlation coefficients between the Japanese HCAHPS composites and overall hospital rating exceeded the criteria. Results of inter-item correlations indicated adequate internal consistency reliability.
CONCLUSIONS: The Japanese HCAHPS has acceptable psychometric properties for assessing patients' experience with hospital inpatient care. This scale could be used for quality improvement based on the assessment of patients' experience with hospital care and for health services research in Japan. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The Japanese Hospital Consumer Assessment of Healthcare Providers and Systems is the first validated scale measuring patients’ experience with hospital inpatient care in Japan.

Our data were collected from a large number of hospitals that were distributed widely throughout Japan and covered various hospital sizes and regions.

Although we examined the structural validity, criterion-related validity and internal consistency reliability of the Japanese version developed in this study, other psychometric properties, including convergent and discriminant validity, test–retest reliability and interpretability, have not been assessed.

Introduction

In recent years, better patients’ perceptions of quality of healthcare have been deemed as one of the crucial goals of healthcare. Thus, patient experience has been globally considered as an important quality indicator in a wide range of settings.1 2 Patient experience is integrally tied to the principles and practices of patient-centred and family-centred care. Embedded within patient experience is a focus on individualised care and tailoring services to meet patients’ needs and engage them as partners in their care.3 Patient experience has recently replaced patient satisfaction because there are some limitations regarding the assessment of patient satisfaction, such as poor discriminability.4 Several studies have shown that patient experience is consistently positively associated with clinical effectiveness, patient safety and patient behaviours (such as adherence to medication, use of screening services and resource use) across a wide range of disease areas, settings, population groups and outcome measures.5–8

The Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) is a well-established and internationally recognised scale for measuring patients’ experience with hospital care.9 This scale was developed by the Centres for Medicare and Medicaid Services (CMS) in partnership with and funded by the Agency for Healthcare Research and Quality (AHRQ).10 In the USA, HCAHPS results have been linked to financial reimbursement from Medicare and other insurers for promoting quality improvement in hospitals.11 Additionally, these results are posted on the website for helping patients’ decision-making process by enabling comparisons across hospitals.12

In Japan, activities for assessment of patient experience have just begun in limited settings, and systematic approaches for quality improvement based on patients’ perceptions of healthcare are still inadequate. In recent years, several scales have been developed and validated to assess outpatients’ experience, mainly in the primary care setting.13–15 However, there are no validated scales for assessing patients’ experience with hospital inpatient care in Japan. Accordingly, the present study aimed to develop a Japanese version of the HCAHPS and to examine its structural validity, criterion-related validity and internal consistency reliability.

Methods

Design, setting and participants

This multicentre cross-sectional study was conducted in 48 hospitals from September to December 2019, in cooperation with the Nihon Hospital Alliance (NHA), which is a group purchasing organisation in Japan. Since 2014, the NHA has conducted an annual patient experience survey to evaluate and improve patient centeredness in hospitals in Japan16; however, the patient experience scale used by the NHA was not validated before. The participating hospitals voluntarily participated in the present study. Table 1 shows the characteristics of the participating hospitals. These hospitals were distributed widely throughout Japan, covering both urban and rural areas. The majority of the hospitals were large (≥400 beds), publicly owned, general hospitals and they had an intensive care unit (ICU). Table 1 also shows the comparison of hospital characteristics between the participating hospitals and hospitals across Japan. We noted a trend suggesting that the proportions of large hospitals and publicly owned hospitals were higher in participating hospitals compared with hospitals across Japan. A self-administered questionnaire was distributed to patients aged ≥16 years who were discharged from the participating hospitals during the survey period. In each participating hospital, eligible participants were selected using a continuous sampling method until reaching the target number of patients (300–600 patients) according to the hospital size. Patients who were unable to respond to the questionnaire due to severe physical or mental disorders were excluded. We collected completed surveys by mail.

Table 1

Characteristics of the 48 participating hospitals and comparison with hospitals across Japan

Characteristic	n (%)
	Participating hospitals(N=48)	Hospitals across Japan*(N=8372)
Hospital size
Small (<200 beds)	4 (8.3)	5790 (69.2)
Medium (200–399 beds)	17 (35.4)	1794 (21.4)
Large (≥400 beds)	27 (56.3)	788 (9.4)
Ownership
Public	34 (70.8)	1583 (18.9)
Private	14 (29.2)	6789 (81.1)
Hospital type
General hospital	46 (95.8)	–
Special hospital	2 (4.2)	–
ICU
Yes	29 (60.4)	–
No	19 (39.6)	–
Hospital region
North	10 (20.8)	1147 (13.7)
East	20 (41.7)	3307 (39.5)
West	18 (37.5)	3825 (45.7)
Okinawa	0 (0.0)	93 (1.1)
Municipality population size
Small (<50 000)	6 (12.5)	–
Medium (50 000–200 000)	14 (29.2)	–
Large (>200 000)	28 (58.3)	–

*Survey of Medical Institutions 2018 by Ministry of Health, Labour and Welfare.

ICU, intensive care unit.

Measures

The HCAHPS

The original HCAHPS is a 19-item tool comprising 6 composites, 2 global ratings and 3 screening items.9 The composites are communication with nurses (Q1–Q3), communication with doctors (Q5–Q7), responsiveness of hospital staff (Q4 and Q11), hospital environment (Q8 and Q9), communication about medicines (Q13 and Q14) and discharge information (Q16 and Q17). The global ratings include overall hospital rating (Q18) and willingness to recommend the hospital to friends and family (recommended hospital) (Q19).

We obtained permission for translating the HCAHPS into Japanese from the AHRQ and CMS. According to the guidelines for translating CAHPS surveys provided by the AHRQ.17 This translation approach involves using two translators to each produce a forward translation and then having the two forward translations reviewed against each other and compared with the original English survey. Translation of the HCAHPS into Japanese was conducted through the following steps, which is common to the development of the Japanese CAHPS Clinician & Group Survey.18 First, two forward translations from English to Japanese were performed independently by two bilingual translators who had prior professional experience in translating survey instruments for health services. Subsequently, the two forward translations were reviewed by a translation reviewer, who was a native speaker of Japanese and had prior experience in translating survey instruments. After reviewing the translations, the reviewer produced a reconciled version of the translation. The final version of the translation was then produced through discussion in a committee composed of the two translators and the reviewer. The reconciled version from the original review was modified as needed based on the committee’s decision for cross-cultural adaptation. The final wording of each survey item and response option was determined by consensus (online supplemental file).

The HCAHPS survey uses several different response scales: a dichotomous scale (1=yes, 2=no), a global rating scale (0=worst to 10=best) and 4-point Likert scales (1=never, 2=sometimes, 3=usually and 4=always; and 1=definitely no, 2=probably no, 3=probably yes and 4=definitely yes). To make the results easier to understand, we converted all scales to normalised scores ranging from 0 to 100 using the following formula:

Normalised score=100*(respondent’s selected response value−minimum response value on the scale)/(maximum response value−minimum response value)

In the Japanese version, assuming the convergence in each composite as in the original version, the score for each of the six composites was computed as the mean value for all normalised scores in the scale that would fall in the range of 0–100 points, with higher scores indicating better performance.

Statistical analysis

To validate the Japanese HCAHPS, we first conducted a confirmatory factor analysis to evaluate the structural validity of the Japanese HCAHPS composites. In the factor analysis, we hypothesised the same factor structure (six-factor solution) as that of the original HCAHPS. The appropriateness of the resulting structure was determined by examining if factor loadings were 0.40 or greater.19 The model fitness was assessed by the comparative fit index (CFI), Tucker-Lewis index (TLI), root mean square error of approximation (RMSEA) and standardised root mean square residual (SRMR). Guidelines suggest that models with CFI and TLI close to 0.95 or higher, RMSEA close to 0.06 or lower, and SRMR close to 0.08 or lower are representative of models with a good fit.20

Subsequently, the Japanese HCAHPS composite scores and the overall hospital rating were used to examine criterion-related validity. Validity was assessed using Pearson correlation coefficients with each Japanese HCAHPS composite to predict the overall hospital rating at the hospital level. A correlation coefficient greater than 0.30 was considered meaningful.21 Hospital-level correlations are a more important criterion for measurement than are individual-level correlations because the former are benchmarking tools to compare one hospital with another. To examine hospital-level correlations, we used each hospital’s mean score on HCAHPS composites and the overall hospital rating.

The internal consistency reliability was examined by inter-item correlations and the Cronbach’s alpha. For a scale to be considered sufficiently reliable, an inter-item correlation of 0.30 and a Cronbach’s alpha value of 0.70 is recommended.22 Finally, descriptive statistics were performed for the Japanese HCAHPS scores, including the mean, SD and observed range. To deal with missing data, in the confirmatory factor analysis, we used full information maximum likelihood estimation to enable the use of information collected from participants with missing data. In the evaluation of criterion-related validity and internal consistency, we conducted complete case analyses. All statistical analyses were conducted using R V.3.6.3 (R Foundation for Statistical Computing, Vienna, Austria; www.R-project.org).

Results

Of the total 15 512 eligible participants, 6522 (42.0%) responded to the survey. Table 2 shows the participants’ responses to each item of the Japanese HCAHPS. The Top Box score for each item, which is the percentage of participants who provided the most positive responses on that item, ranged from 31.1% to 82.5%. Regarding the mean Top Box score for composites, the highest score was observed for discharge information (77.2%), whereas the lowest score was for the hospital environment (49.1%). The bottom box score, which is the percentage of participants with the least positive responses on the item, ranged from 0.5% to 23.2%.

Table 2

Response to Japanese HCAHPS items (N=6522)

	n (%)
	Never	Sometimes	Usually	Always	Data missing	Not applicable
Communication with nurses
Q1. During this hospital stay, how often did nurses treat you with courtesy and respect?	49 (0.8)	196 (3.0)	1986 (30.5)	4263 (65.4)	28 (0.4)	–
Q2. During this hospital stay, how often did nurses listen carefully to you?	34 (0.5)	257 (3.9)	2075 (31.8)	4127 (63.3)	29 (0.4)	–
Q3. During this hospital stay, how often did nurses explain things in a way you could understand?	42 (0.6)	272 (4.2)	2259 (34.6)	3915 (60.0)	34 (0.5)	–
Communication with doctors:
Q5. During this hospital stay, how often did doctors treat you with courtesy and respect?	50 (0.8)	179 (2.7)	1549 (23.8)	4525 (69.4)	219 (3.4)	–
Q6. During this hospital stay, how often did doctors listen carefully to you?	59 (0.9)	232 (3.6)	1701 (26.1)	4291 (65.8)	239 (3.7)	–
Q7. During this hospital stay, how often did doctors explain things in a way you could understand?	57 (0.9)	283 (4.3)	1735 (26.6)	4207 (64.5)	240 (3.7)	–
Responsiveness of hospital staff:
Q4. During this hospital stay, after you pressed the call button, how often did you get help as soon as you wanted it?	50 (1.0)	213 (4.1)	1467 (28.1)	3429 (65.7)	64 (1.2)	1299
Q11. How often did you get help in getting to the bathroom or in using a bedpan as soon as you wanted?	33 (1.8)	109 (5.8)	602 (32.1)	1050 (56.0)	81 (4.3)	4647
Hospital environment:
Q8. During this hospital stay, how often were your room and bathroom kept clean?	46 (0.7)	217 (3.3)	1940 (29.7)	3966 (60.8)	353 (5.4)	–
Q9. During this hospital stay, how often was the area around your room quiet at night?	187 (2.9)	731 (11.2)	2775 (42.5)	2438 (37.4)	391 (6.0)	–
Communication about medicines:
Q13. Before giving you any new medicine, how often did hospital staff tell you what the medicine was for?	95 (2.9)	145 (4.4)	762 (23.2)	2238 (68.0)	51 (1.5)	3231
Q14. Before giving you any new medicine, how often did hospital staff describe possible side effects in a way you could understand?	399 (12.1)	305 (9.3)	1016 (30.9)	1490 (45.3)	81 (2.5)	3231
	No	Yes	Data missing	Not applicable
Discharge information:
Q16. During this hospital stay, did doctors, nurses or other hospital staff talk with you about whether you would have the help you needed when you left the hospital?	838 (14.3)	4846 (82.5)	190 (3.2)	648
Q17. During this hospital stay, did you get information in writing about what symptoms or health problems to look out for after you left the hospital?	1365 (23.2)	4222 (71.9)	287 (4.9)	648
	0–2	3–5	6–8	9–10	Data missing	Not applicable
Overall hospital rating:
Q18. Using any number from 0 to 10, where 0 is the worst hospital possible and 10 is the best hospital possible, what number would you use to rate this hospital during your stay?	44 (0.7)	291 (4.5)	2238 (34.3)	3591 (55.1)	358 (5.5)	–
	Definitely no	Probably no	Probably yes	Definitely yes	Data missing	Not applicable
Recommended hospital:
Q19. Would you recommend this hospital to your friends and family?	53 (0.8)	300 (4.6)	3772 (57.8)	2027 (31.1)	370 (5.7)	–

Not applicable: the number of participants who skipped the item due to the response to the screenig item.

HCAHPS, Hospital Consumer Assessment of Healthcare Providers and Systems.

Structural validity

Figure 1 shows the path diagrams of the confirmatory factor analysis to assess the structural validity of the Japanese HCAHPS composites. All factor loadings of each item onto each factor were above the 0.40 criteria, ranging from 0.41 to 0.91. The correlation coefficients among factors ranged from 0.30 to 0.85. The conceptual model showed excellent goodness of fit, with CFI=0.987, TLI=0.981, RMSEA=0.031 and SRMR=0.020.

Figure 1

Factor structure of Japanese HCAHPS (confirmatory factor analysis). Squares are observed variables (items); ellipses are latent variables (factors), values on the single-headed arrows are standardised factor loadings, values on the double-headed arrows are correlation coefficients. HCAHPS, Hospital Consumer Assessment of Healthcare Providers and Systems.

Criterion-related validity

Table 3 shows the Pearson correlation coefficients between the Japanese HCAHPS composites and the overall hospital rating at the hospital level. All correlations were statistically significant (p<0.01), and they exceeded the 0.30 criterion. The composite ‘communication with doctors’ (r=0.63) had the highest correlation with the overall rating.

Table 3

Pearson correlation coefficients between Japanese HCAHPS composites and overall hospital rating

Composites	Hospital-level correlations
Communication with nurses	0.62
Communication with doctors	0.63
Responsiveness of hospital staff	0.36
Hospital environment	0.56
Communication about medicines	0.58
Discharge information	0.41

HCAHPS, Hospital Consumer Assessment of Healthcare Providers and Systems.

Internal consistency reliability and descriptive statistics

Table 4 indicates the score distribution and internal consistency reliability for the Japanese HCAHPS. All inter-item correlations were above the 0.30 criteria, ranging from 0.31 to 0.73. For communication with nurses, communication with doctors and communication about medicines, the Cronbach’s alpha was above 0.70. In contrast, for the responsiveness of hospital staff, hospital environment and discharge information, the Cronbach’s alpha did not exceed the 0.70 criterion. Descriptive statistics showed that the highest scored scale was communication with doctors (mean score=87.9), and the lowest scored scale was recommended hospital (mean score=75.5). The full range of possible scores was observed for all scales.

Table 4

Descriptive features and internal consistency reliability of Japanese HCAHPS (N=6522)

	Number of items	Mean	SD	Observed range	Inter-item correlation	Cronbach’s alpha
Composites
Communication with nurses	3	86.1	17.4	0.0–100.0	0.62–0.73	0.85
Communication with doctors	3	87.9	18.1	0.0–100.0	0.65–0.70	0.89
Responsiveness of hospital staff	2	81.0	21.1	0.0–100.0	0.34	0.46
Hospital environment	2	80.2	18.7	0.0–100.0	0.35	0.49
Communication about medicines	2	78.7	25.8	0.0–100.0	0.61	0.71
Discharge information	2	80.3	32.0	0.0–100.0	0.31	0.48
Global ratings
Overall hospital rating	1	85.7	15.4	0.0–100.0	–	–
Recommended hospital	1	75.5	19.5	0.0–100.0	–	–

HCAHPS, Hospital Consumer Assessment of Healthcare Providers and Systems.

Discussion

Measurement of patient experience plays an important role in the improvement of a wide range of medical services, including inpatient care. We translated the HCAHPS, which is a validated international scale, into Japanese and examined its structural validity, criterion-related validity and internal consistency reliability in 48 hospitals in Japan. This study was the first to develop a Japanese version of the HCAHPS and to examine its psychometric properties for assessing patients’ experience with hospital inpatient care.

Standard psychometric evaluation methods were used to evaluate the Japanese HCAHPS. The confirmatory factor analysis supported the scale’s structural validity and the same six-factor solution as that of the original HCAHPS, with good statistical fitness. However, some items, such as Q11, had relatively lower factor loadings. Thus, the scale’s structural validity might need to be confirmed in other settings. Correlation coefficients between all Japanese HCAHPS composites and the overall hospital rating for assessing criterion-related validity exceeded the meaningful value at the hospital level. However, the responsiveness of hospital staff and discharge information composites had relatively lower correlations with the overall hospital rating. Other reference scales should also be used to examine the scale’s criterion-related validity in further studies.

In internal consistency analyses, the Cronbach’s alpha for the responsiveness of hospital staff, hospital environment and discharge information did not exceed the optimum criterion. The Cronbach’s alpha is quite sensitive to the number of items in the scale; therefore, it is common to find low Cronbach’s alpha for scales with few items (especially 2-item scales).23 Likewise, a study conducted in the USA24 found that the Cronbach’s alpha was low for some aspects of the original HCAHPS scale. In this case, it is more appropriate to report the inter-item correlation of items. In our study, all inter-item correlations were greater than the criterion, which indicated adequate internal consistency of the scales.

To our knowledge, the Japanese HCAHPS is the first validated scale measuring patients’ experience with hospital inpatient care in Japan. The HCAHPS is one of the most widely studied and endorsed patient experience measure of hospital care worldwide. The HCAHPS items and composites are considered to be suitable for the Japanese healthcare system as they are included in the hospital accreditation standards in Japan.25 Our data were collected from a large number of hospitals that were distributed widely throughout Japan and they covered various hospital sizes and regions. Therefore, the study results have relatively high external validity.

However, there are several potential limitations to our study. First, the response rate was a concern. A previous study of patient experience surveys showed that a low participation rate is less likely to introduce selective non-response bias26; however, it is possible that patients with worse experience were less likely to respond to our survey. Second, although we examined the structural validity, criterion-related validity and internal consistency reliability of the Japanese HCAHPS in this study, other psychometric properties, including convergent and discriminant validity, test–retest reliability and interpretability have not been assessed.27 These measurement properties of the scale need to be evaluated in future studies. Third, this study was limited by the fact that the participating hospitals voluntarily participated in this study; thus, the preset sample may represent hospitals that have a higher interest in the quality of healthcare. In addition, there were some differences in hospital characteristics between the participating hospitals and hospitals across Japan. Accordingly, the participating hospitals may not have sufficiently represented Japanese hospitals at the national level. Therefore, Japanese HCAHPS should be used for research in other settings.

Conclusion

The Japanese HCAHPS has acceptable psychometric properties for assessing patients’ experience with hospital inpatient care. This scale could be used for quality improvement based on the assessment of patients’ experience with hospital care and for health services research in Japan.