A Psychometric Analysis of the Health Literate Health Care Organization-10 Item Questionnaire.

The concept of a Health Literate Healthcare Organization (HLHO) is a relatively new approach to health literacy that moves the focus from the individual patient to the overarching health care system. The HLHO-10 questionnaire was developed internationally to assess the 10 Attributes of HLHOs as described by participants of the Institute of Medicine Roundtable on Health Literacy. The purpose of this study was to establish reliability and validity of the HLHO-10 among a sample of United States hospitals. Reliability and validity were established through assessing the factor structure for the HLHO-10 and psychometric evaluation. The HLHO-10 was found to be reliable with a Cronbach's alpha of .855 and a two-factor structure was revealed through exploratory factor analysis. Additional research is needed to further validate use of the HLHO-10 in the U.S., but initial findings of this emerging tool are promising and timely as the issue of health literacy comes to the forefront of U.S. health care systems and associated regulatory agencies. [HLRP: Health Literacy Research and Practice. 2022;6(2):e137-e141.].

Health literacy remains an enormous challenge for health care providers and patients alike. Originally conceived as an individual patient's skill or lack of skill, the focus was on health literacy screening (Wolf et al., 2007). More recently, health literacy experts have broadened the concept of health literacy to a function of the interaction between health care providers, the health system, and the patient (Koh et al., 2013). This perspective is emphasized in Healthy People 2030, which defines organizational health literacy as “the degree to which organizations equitably enable individuals to find, understand, and use information and services to inform health-related decisions and actions for themselves and others” (Office of Disease Prevention and Health Promotion [ODPHP], 2020, para. 5). The inclusion of organizational health literacy in addition to personal health literacy provides a public health perspective while emphasizing the responsibility of healthcare organizations to address and promote health literacy (ODPHP, 2020).

Participants from the Institute of Medicine (IOM) Roundtable on Health Literacy established 10 aspirational attributes of the health literate organization that make it easier for patients to understand and use health information to take care of their health (Brach et al., 2012). These 10 attributes became the foundation for establishing organizational health literate practices. Kowalski et al. (2015) developed the 10-item Health Literate Health Organization questionnaire (HLHO-10) to assess the extent the 10 HLHO attributes are implemented within health care organizations. Each questionnaire item reflects one of the IOM Roundtable's 10 HLHO attributes (Table ). Respondents answer using a seven-point Likert scale ranging from not at all to a very large extent. Instrument development and psychometric analysis of the HLHO-10 was conducted in Germany with key informants from 51 breast cancer center hospitals (Kowalski et al., 2015). The Cronbach's alpha of the HLHO-10 in this population was 0.89. Confirmatory factor analysis confirmed a one factor model (Kowalski et al., 2015).

After inception and validation, the HLHO-10 began to be used around the world. Prince et al. (2018) assessed organizational health literate practices at a United States academic health center but did not provide psychometric analysis. Hayran and Ozer (2018) evaluated health literate practices across hospitals in Istanbul, Turkey. The Cronbach's Alpha was reported as 0.916. Results demonstrated criterion validity as a significant association between patient satisfaction and HLHO-10 scores. Bonaccorsi et al. (2020) translated the HLHO-10 into Italian and validated the instrument within the Italian health care system. Exploratory factor analysis suggested multidimensionality with three factors emerging from the analysis. These results differ from the original validation of the HLHO-10 (Bonaccorsi et al., 2020).

Although the HLHO-10 has been used in research internationally and statistically analyzed in the United States, at the time of writing no literature exists where the HLHO-10 has been psychometrically evaluated in the U.S. As the desire for health literate organizations gains traction in both importance and prevalence within health care, this emerging instrument has the potential to become an important tool. The purpose of this study was to establish reliability and validity of the HLHO-10 among hospitals in the U.S. Reliability and validity will be established through assessing the factor structure for the HLHO-10 and psychometric evaluation.

Methods

The HLHO-10, as reported by Kowalski et al. (2015) in English, was used by Howe et al. (2020) as part of a larger mixed methods study to describe the perceptions about organizational health literate policies and practices. After Institutional Review Board approval, the HLHO-10 was distributed to key informants from 13 hospitals across 5 health care systems in North Texas. More than one individual from each organization was surveyed to capture the depth and breadth of each organization from multiple perspectives. In total, data from the HLHO-10 questionnaire and qualitative interviews were collected from 74 participants from 12 hospitals with roles of frontline team members, middle managers, and executive leaders. The design, sampling methods, and procedures of the larger study have been described elsewhere (Howe et al., 2020).

Results

Data Screening

The data were screened for univariate outliers. No outof-range values were identified. Although the sample size is small for conducting exploratory factor analysis, according to Thompson's (2004) Exploratory and Confirmatory Factor Analysis, the minimum amount of data for factor analysis was satisfied, with a final sample size of 74 and ratio of over seven cases per variable.

Descriptive Statistics and Inter-rater Reliability Analysis

Descriptive statistics of the HLHO-10 survey are presented by organization in Table . As multiple respondents participated from each hospital, inter-rater reliability analysis was conducted on the sample to assess responses both within and between organizations. Intraclass correlation coefficients (ICC) estimates and their 95% confident intervals were calculated using IBM's SPSS 26 software based on a mean-rating, consistency, 2-way random-effects model. ICC for every hospital was analyzed and each had good inter-rater reliability. Inter-rater reliability amongst all 74 subjects on the HLHO-10 was also good (ICC = 0.855, 95% confidence interval [0.8, 0.9]).

Factor Analysis

Initial factorability of the 10 HLHO items was assessed using well-recognized criteria. An inspection of the Pearson's r correlation matrix showed all coefficients above 0.5, the Kaiser-Meyer-Olkin measure of sample adequacy was 0.833, exceeding the recommended value of 0.6, and the Bartlett's Test of Sphericity was statistically significant (p < .001), all of which support the factorability of the correlation matrix. Given these overall indicators, factor analysis was deemed to be suitable with all 10 items.

The HLHO-10 was subjected to Principal Axis Factoring (PAF) with a Direct Oblimin rotation using SPSS. PAF was chosen for structure detection and to analyze covariance of the items. Factors with eigenvalues greater than 1.00 were retained; PAF revealed a two-factor structure. Factor loadings are reported in Table . Two items cross-loaded onto both factor 1 and 2: item number 8 (“media”), loaded at .389 and .372 and item number 9 (“communication in high-risk situations”) loaded at .528 and .406. Eight items loaded on the first factor, categorized as items with clinical operations oversight, and two items loaded on the second factor, categorized as items with nonclinical administrative oversight (see Table ). However, upon further discussion, the decision was made to include item number 8 (“media”) on factor 2 for conceptual alignment. The first factor explained 46.56% of variance, which was almost four times that of the variance explained by the second factor (13.25%), suggesting the existence of a dominant latent factor.

The internal consistency for the HLHO-10 was strong (α = .855). Once the factors were established, Cronbach's alphas were computed to determine the internal consistency of each factor. The internal consistency for the first factor, items with clinical operations oversight, was strong (α = .865) and low for the second factor, items with non-clinical administrative oversight, (α = .617) (Tavakol & Dennick, 2011).

Discussion

Overall, the psychometric analysis produced positive results for both the validity and reliability for use of the HLHO-10 scale in the U.S. There is consensus among the authors that the items strongly loaded on factor 1 adequately assessed the associated IOM Roundtable's attributes and measured overarching health literacy of healthcare organizations. Likewise, there was agreement that these factors were generally clinically oriented, concrete, and managed at the operations level by front-line clinical leadership within a particular organization. In contrast, the items that loaded solely on factor 2 (access and navigation and costs) were viewed as more abstract health care services managed at the administrative level by nonclinical leadership within an organization with influence from regional, state, and federal regulations. Item number 8 (“media”) cross-loaded closely between the two factors and after much consideration, was included on factor 2 for conceptual alignment, as oversight for hospital media generally lies at a larger, nonclinical, administrative level. These findings conflict with initial findings by Kowalski et al. (2015) but are consistent with Bonaccorsi et al. (2020), whose third dimension included access & navigation, costs, and media, and was described as items that refer to navigation within the hospital and communication with patients regarding healthcare costs.

Kowalski et al. (2015) acknowledge the IOM Roundtable's attributes were simplified in the development of the HLHO-10 scale to promote relevance and understanding within hospital key informants (see Table ). Likewise, they note the development was focused on the patient-centered healthcare system of Germany and presumably was translated from German to the English language for publication. In analyzing our findings and assessing the qualitative data collected in the mixed methods approach of Howe et al. (2020), we noted potential challenges with the translation of the IOM Roundtable's attributes into HLHO-10 scale items for use in the U.S., particularly for those questions that loaded on to factor 2. For example, item 7 (“access and navigation”) asks the degree to which patients can find their way in the hospital with signage and informational staff, whereas the IOM Roundtable's attribute describes this as the broader concept of navigation of the health care system (Brach et al., 2012).

Brach et al. (2012) acknowledge the field of health literacy is young and state the 10 attributes “provide. . . an aspirational vision” while warning “the road to becoming health literate. . . is a long one” (p. 2). The initial goal of the IOM Roundtable's attributes is to provide guideposts and this perspective must be maintained as we embark on future work with the HLHO-10 scale. While the items that loaded on factor 1 are straightforward in their relationship to health literacy, intentional efforts for engagement of both internal and external stakeholders will be critical to address the items with nonclinical administrative oversight that compose factor 2. A statistical argument could be made to remove the factor 2 items from the HLHO-10 scale in future research, but the authors agree that philosophically these concepts are fundamental to organizational health literacy as outlined by the IOM Roundtable.

Regulatory requirements unique to the U.S. add an additional layer of complexity in using the HLHO-10 to assess health literacy in healthcare organizations. The Centers for Medicare & Medicaid Services (CMS) promotes guidelines (notably not requirements) on the importance of health literacy (CMS, 2020) and The Joint Commission, which accredits the majority of healthcare organizations in the United States has integrated health literacy into regulatory standards, particularly around effective communication (Institute of Medicine Roundtable on Health Literacy, 2009). In addition, the 21st Century Cures Act is undoubtedly casting a new focus on transparency of healthcare costs, access, and outcomes (Department of Health and Human Services, 2020) that is requiring healthcare organizations to pivot in ways not previously required and subsequently bringing health literacy to the fore-front of health care delivery.

Building on the prior work completed by our international colleagues, further testing of the HLHO-10 scale is needed. This work should focus on elucidating challenges with translation of the IOM Roundtable's attributes, potentially adapting language more specific to American practices for future use in the U.S. and determining if item refinement is indicated versus identification of true challenges in organizational health literacy that will require institutional, regional, state, and federal collaboration to overcome.

Study Limitations

Limitations of this study include a small sample size and limited geographic variability. The concept of health literate health care organizations is relatively new and rich with potential for further research to include a broader reach across the U.S. with larger samples of participating hospitals. There is also great potential to expand the use of the HLHO-10 instrument to compare health literacy practices and assess interventional effectiveness over time both within and across hospitals and to evaluate overarching organizational accountability to health literacy as emphasized in Healthy People 2030.

Table 1

IOM Roundtable's Attributes and HLHO-10 Items[a]

Item	IOM Roundtable's Attribute	HLHO-10 Item Questionnaire: To What Extent …
1. Leadership	Has leadership that makes health literacy integral to its mission, structure, and operations	Is the management at your hospital explicitly dedicated to the subject of health literacy (e.g., mission statement, human resource planning)?
2. Integration	Integrates health literacy into planning, evaluation measures, patient safety, and quality improvement	Is the topic of health literacy considered in quality management measures at your hospital?
3. Workforce training	Prepares the workforce to be health literate and monitors progress	Are employees at your hospital trained on the topic of health literacy?
4. Patient inclusion	Includes populations served in the design, implementation, and evaluation of health information and services	Is health information at your hospital developed by involving patients?
5. Health literacy skills	Meets the needs of populations with a range of health literacy skills while avoiding stigmatization	Is individualized health information used at your hospital (e.g., different languages, print sizes, braille)?
6. Communication standards	Uses health literacy strategies in interpersonal communications and confirms understanding at all points of contact	Are there communication standards at your hospital that ensure that patients truly understand the necessary information (e.g., translators, allowing pauses for reflection, calling for further queries)?
7. Access and navigation	Provides easy access to health information and services and navigation assistance	Are efforts made to ensure that patients can find their way at your hospital without any problems (e.g., direction signs, information staff)?
8. Media	Designs and distributes print, audiovisual, and social media content that is easy to understand and act on	Is information made available to different patients via different media at your hospital (e.g., three-dimensional models, DVDs, picture stories)?
9. Communication in high-risk situations	Addresses health literacy in high-risk situations, including care transitions and communications about medicines	Is it ensured that the patients have truly understood everything, particularly in critical situations (e.g., medication, surgical consent) at your hospital?
10. Costs	Communicates clearly what health plans cover and what individuals will have to pay for services	Do you communicate openly and comprehensibly at your hospital to your patients in advance about the costs that they themselves have to pay for treatment (e.g., out-of-pocket payments)?

Note. IOM = Institute of Medicine; HLHO-10 = Health Literate Health Care Organization-10 Item Questionnaire.

This table compares the 10 attributes of HLHO (Brach et al., 2012) with the HLHO-10 Item Questionnaire developed by Kowalski et al., 2015.

Table 2

Descriptive Statistics of Total HLHO-10 Score by Institution

Institution	Number	M	SD	Range
1	5	40	7.616	20
2	6	43.5	4.087	10
3	6	40.17	11.125	28
4	8	45.63	10.9	30
5	8	42.63	8.959	22
6	5	45.6	12.482	31
7	7	47.57	11.83	36
8	4	45	13.711	30
9	6	59.5	7.45	19
10	10	44.9	11.976	35
11	6	45.67	7.501	19
12	3	46.67	3.786	7
All	74	45.53	10.377	41

Note. HLHO-10 = Health Literate Health Care Organization-10 Item Questionnaire; M = Mean; SD = standard deviation.

Table 3

Rotated Factor Loadings for HLHO-10 (N = 74)[a]

Factor Name and Item	Factor 1	Factor 2
Clinical operations
Integration (2)	.872
Leadership (1)	.794
Communication standards (6)	.647
Patient inclusion (4)	.620
Workforce training (3)	.578
Health literacy skills (5)	.568
Communication in high-risk situations (9)	.528	.406
Nonclinical administrative operations
Media (8)	.389	.372
Access and navigation (7)		.814
Costs (10)		.466

Note. Bold values indicate highest loading. HLHO-10 = Health Literate Health Care Organization-10 Item Questionnaire.

Extraction method: Principal axis factoring. Rotation method: Direct Oblimin with Kaiser Normalization.