Assessing the Print Communication and Technology Attributes of an Academic Medical Center.

BACKGROUND: Historically, health literacy has been viewed as the patient's problem; however, it is now accepted that the responsibility for improving health literacy lies with the health care professionals and systems. An Institute of Medicine report outlines the health literacy attributes, such as printed patient education and technology, which both play a role in patient decision-making and engaging them in their health care. Research suggests that patients who are engaged in their health care have improved health outcomes. For health care organizations to accommodate the needs of all patients, it is imperative that they determine the current organizational state and discover opportunities for improvement.
METHODS: The Health Literacy Environment of Hospitals and Health Centers (HLEHHC) Print Communication Rating and Technology Rating Tool were used to measure the internal aspects of organizational health literacy at The University of Tennessee Medical Center (UTMC). Included in the print assessment were the 150 most distributed patient education handouts. Researchers also used the Simple Measure of Gobbledygook and Patient Education Materials Assessment Tool to assess print material. Technology was assessed using UTMC's website as the authoritative source. KEY
RESULTS: The HLEHHC was useful for assessing print material and technology. Reviewing and reporting the data question-by-question revealed more granular, actionable information on where there are opportunities to improve the health care environment for all patients. This analysis resulted in proposing actions based on best practices that UTMC could implement in the coming year. The process is replicable in other settings. IMPLICATIONS: Responsibility for improving informed medical decision-making lies with health care organizations. Low health literacy influences the effectiveness of print patient education and technology in informing patients about their health. Assessing these aspects of the health care organization as part of quality improvement provides necessary data for improvements. The Health Literacy Environment of Hospital and Health Centers was a useful tool to measure characteristics of print and technology. [HLRP: Health Literacy Research and Practice. 2018;2(1):e26-e34.]. PLAIN LANGUAGE
SUMMARY: A task force at an academic medical center assessed the health literacy attributes of their organization. Researchers assessed print patient education and patient-related technology. The researchers found areas for improvements to make health information easier to understand.

In the past, health literacy was viewed as a problem for patients and that it was their burden to acquire the necessary skills to understand and make decisions about their health. However, as illustrated in the literature (Koh, Brach, Harris, & Parchman, 2013; Koh & Rudd, 2015; Parker, Ratzan, & Lurie, 2003), it is now understood, and generally accepted, that the primary responsibility for improving health literacy lies with the health care professionals and systems.

The National Action Plan to Improve Health Literacy recognized the health literacy problem in the United States and focused on systematic issues in the health care system rather than on the shortcomings of patients (U.S. Department of Health and Human Services, 2010). The National Action Plan put forth seven goals to restructure how health education is conducted and how health information is disseminated. The second goal of the National Action Plan called on health care organizations to “Promote changes in the health care delivery system that improve health information, communication, informed decision making, and access to health services” (U.S. Department of Health and Human Services, 2010). Subsequent to the report, a set of criteria by which organizations could gauge whether they made it “easier for people navigate, understand, and use information and services to take care of their health” was created (Brach et al., 2012).

Among the focal points of these two documents are several references to health materials, including print patient education, paper forms, audiovisual materials, and technology, such as patient portals, touch screens, social media content, and blogs. Each of these materials has the potential to inform patients about their health care. “Informed medical decision-making,” a term introduced by McNutt in 2004, best describes this process, and research suggests that patients who are engaged in their health care have improved health outcomes (Stacey et al., 2017; Weiner et al., 2013). The use of technology and print resources have both been promoted as ways to inform patients about the decisions they must make and better engage them in their health care (Woolf et al., 2005).

As part of a performance improvement process, The University of Tennessee Medical Center (UTMC), located in East Tennessee, formed a task force, led by medical librarians, to assess the current state of the organization in regards to health literacy to provide a basis for promoting changes. This article reports on the assessment of print communication and use of technology, including the assessment tool choice, the research methods, statistical analyses, and results.

Instruments

There are several assessment tools available to assess an organization's health literacy attributes. All of the tools fall into the categories of surveys and checklists, and although they are helpful and easy to apply, none have been truly validated as research tools (Kripalani et al., 2013). Authors of the tools include government and private institutions, both in the U. S. and internationally. The target respondent for these measures is either an organization, individual provider, or patient (Kripalani et al., 2013).

The criteria for choosing an assessment tool to apply to UTMC's setting included the following: that it be based in the U. S. health care system; include organization respondents; used in other health care organizations; and assessed the most health literacy attributes, as defined by the Ten Attributes (Brach et al., 2012).

The Health Literacy Environment of Hospitals and Health Centers (HLEHHC), created by Rudd and Anderson (2006), offered a set of tools to measure five aspects of the health care organization that impact patients with low health literacy. The HLEHHC is not meant for comparison purposes between health care organizations; instead, the tool measures internal aspects of organizational health literacy of one organization. The document is comprised of five categories: Print Communication, Oral Communication, Navigation, Policies and Protocol, and Technology. Each category contains background information for the researcher followed by a series of questions. Respondents answered questions by ranking them as 1, 2, or 3. A ranking of 1 represented “this is something that is not done.” A ranking of 2 represented “this is done, but needs some improvement.” Lastly, a ranking of 3 represented “this is something that is done well.” The resulting aggregate score for each of the five categories was then assigned into three predefined ranges consisting of “begin a focused initiative to eliminate literacy-related barriers,” “augment efforts to eliminate literacy-related barriers,” and “continue to monitor and eliminate literacy-related barriers” (Rudd & Anderson, 2006).

This article discusses the results of the Print Communication and Technology portions of the UTMC's larger HLEHHC assessment project. The HLEHHC for print communication assessed factors that influence how a patient engages with and uses printed material for health decisions. The HLEHHC delineates the complexity of print materials through four distinct sections that highlight areas of influence including: writing style, organization and design, type style (size of print and contrast with paper), and photographs (illustrations, symbols, and diversity). The HLEHHC assessed the use of technology through review of televisions, telephones, patient engagement, website content, and computers.

In addition to using the HLEHHC to assess technology and print, researchers used other methods for assessing the printed patient education material. Assessing the grade level is important as The Joint Commission (2010) recommends that all patient education be at the sixth-grade reading level or below. The HLEHHC recommends using Simple Measure of Gobbledygook (SMOG) to review the grade level of the print material. Rudd and Anderson (2006) state that SMOG is useful for doing quick assessments and predicts 100% comprehension. Based on these recommendations, researchers chose SMOG to review grade level. The HLEHHC recommended Suitability Assessment of Materials (SAM); however, Patient Education Materials Assessment Tool (PEMAT) was deemed the better choice to assess the understandability and actionability of print materials based on recent research demonstrating its validity (Shoemaker, Wolf, & Brach, 2014). Researchers included SMOG and PEMAT in addition to the HLEHHC also to help prepare reviewers to better answer the HLEHHC print communication questions. The research project received an exemption from the Institutional Review Board because there was no identifiable patient information.

Methods

For the print assessment, researchers downloaded the 150 most distributed patient education documents from the hospital's system for review and assigned each document an identification number for tracking and data entry. The 150 pieces of patient education included both materials from ExitCare (a patient education material provider) as well as custom materials, which were created by UTMC staff. Materials were excluded if they were no longer available through ExitCare or if they were only charts or images with no text content. Each document was assessed by three independent reviewers using SMOG, PEMAT, and the Print Communication Rating (PCR) form of the HLEHHC.

Six graduate nursing students, as well as two masters' degree students in public health and counseling were selected as reviewers to complete the print assessment. Each patient education document was reviewed three times by three different reviewers. Reviewers were randomly assigned materials. A medical librarian trained in all three assessment tools provided reviewers with an overview of health literacy and principles of examining easy-to-read materials based on the National Network of Libraries of Medicine's class “Promoting Health Literacy Through Easy-to-Read Materials” (Ottosen, 2015). During training, reviewers practiced applying the tools using documents that were not included in the study. To avoid bias that final PCR scores might have on reviewers, PCR forms were returned to researchers un-totaled. Researchers then totaled the scores for the PCR and entered the data into a spreadsheet for further data analysis.

Frequency statistics were conducted on all variables to check for data entry errors. Skewness and kurtosis statistics were run on continuous variables to assess normality. Independent sample t-tests were used to compare groups on normally distributed continuous variables. Mann-Whitney U tests were employed for outcomes that were not normally distributed. Means, medians, interquartile ranges, standard deviations, and 95% confidence intervals (CI) were reported and analyzed. Pearson's r correlation was used to test associations between continuous variables. Intra-class correlation coefficients (ICC) were used to establish inter-rater reliability for survey instrument ratings. All analyses were conducted using SPSS Version 21 (IBM Corporation; Armonk, NY) and statistical significance was assumed at an alpha value of 0.05.

Researchers evaluated technology use at UTMC using the Technology Rating Tool (TRT) of the HLEHHC. With permission from one of the tool's original authors (R. Rudd, personal communication, September 9, 2016), researchers edited the TRT to better reflect modern-day technology including accessing test results online, accessing prescription history, and requesting health information and video chat from hospital rooms (Table ).

One of the researchers completed the TRT using UTMC's website as the authoritative source. If the website provided answers to questions directly and affirmatively, a rating of “3” was given. If answers were not available on the website, but known to be true by researchers based on experiences and observations, a rating of “2” was given. If answers were not available on the website and were not known to be true, a rating of “1” was given. The data were then entered in a spreadsheet for further analyses. Descriptive statistics were used to explain the prevalence of ratings.

Results

Of the 150 print materials analyzed, 91.3% (n = 137) were original, unedited documents from the ExitCare collection, and 8.79% (n = 13) were custom documents created or edited by UTMC health care providers. All data were normally distributed as determined by skewness and kurtosis statistics.

There was excellent inter-rater reliability between reviewers for the SMOG (ICC = 0.95). SMOG grade level for the combined original and custom was grade 9.6 (95% CI 9.4, 9.8). When analyzed separately, original documents scored at grade 9.6 (95% CI 9.4, 9.9), whereas custom documents scored slightly lower at grade 9.1 (95% CI 8.2, 9.9); however, the difference was not statistically significant (p = .14).

There was good inter-rater reliability for the PCR between reviewers (ICC = 0.67). The mean PCR score for all 150 documents was 53.9 (95% CI 53, 54.9). When comparing original documents to custom, there was a significant difference (p = .02) with a lower score of 50.2 (95% CI 47.6, 52.8) for custom versus 54.3 (95% CI 53.3, 55.3) for original documents.

The researchers evaluated the individual means for each question on the PCR to determine more granular results. In the category of formatting, UTMC scored an average mean of 2.82 to 2.97 of a possible 3. Included in formatting is “font size is 12 points or greater” (µ = 2.96) and “text avoids splitting words across two lines” (µ = 2.97). For the category of visuals and cultural sensitivity, UTMC scored an average mean of 1.39 to 2.74 of a possible 3. Included in this category is that visuals are “representative of the intended audience” (µ = 1.39) and “reinforce key messages” (µ = 1.76). See Figures for a complete list of questions and their aggregate means.

Due to extremely low inter-rater reliability, researchers were unable to run statistics on PEMAT scores. There was poor inter-rater reliability for the Understandability PEMAT (ICC = 0.25) and very poor inter-rater reliability for the Actionability PEMAT (ICC = 0.06). No further results are reported for PEMAT.

The overall technology rating score was 47 of a possible 54. The following is the proportion of rankings on the TRT: 72.2% were ranked as a 3 (highest ranking), 16.7% were ranked as a 2 (middle ranking), and 11.1% were ranked as a 1 (lowest ranking). See Table for details on question rankings.

Discussion

This research is unique in its use of the HLEHHC assessment tools for both print communication and technology. Although Horowitz et al. (2014) and Groene and Rudd (2011) each referenced the HLEHHC in reporting their research, both use it only to assess the navigation of health care settings. Although these authors reported on the assessment of print patient education, neither used the PCR tool, relying instead on SMOG, SAM, or Flesch-Szigriszt. In addition, although the Fox Chase Cancer Center (Philadelphia, PA) reported using the HLEHHC and SMOG to evaluate patient education, researchers did not break down the PCR question by question to find recommendations for further evaluation and research (Raivitch et al., 2010). Finally, Fox Chase Cancer Center (Raivitch et al., 2010), Horowitz et al., 2014, and Groene & Rudd, 2011 used the HLEHHC tools, but did not assess technology.

The process of assessment of the health literacy attributes of a health care setting using the HLEHHC provided rich data, which can be used to make improvements. Overall, UTMC scored well in both Print Communication (53.9 points) and Technology (47 points). The PCR score ranges from 0 to 72, with higher being better and the Technology Rating score ranges from 0 to 54, with higher being better. By breaking down the PCR and the TCR scores question by question, researchers could determine in which areas UTMC scored well and areas with opportunities for improvements. In doing so, researchers learned that for print communication UTMC scored well in the formatting of print patient education, as well as the use of headings, logical grouping of events, and bullets. Areas in which UTMC has opportunities for improvement include cultural sensitivity, use of visuals, and reading grade level.

UTMC scored well in regards to patient engagement through technology because of the availability of bedside televisions to deliver patient education, the ability to request health information from patient rooms, and the availability of computers in more than one location. The organization established an environment for patients to engage. Opportunities for improvement in technology included providing a more engaging patient portal. By examining the aggregate score per question for print communication and technology, researchers could get a specific picture of where UTMC stood and then make recommendations for change to UTMC's senior leadership based on these findings.

Researchers edited the technology tool to reflect today's modern technology. The Centers for Medicare and Medicaid Services emphasize the importance of technology for patients to have the capability to access their health records and, in so doing, become more connected to their provider (Weinstock & Hoppszallern, 2015). The Hospital and Health Network (HHN) awards organizations as the “Most Wired” based on their use of technology to partner with patients on their health (Vesely, 2017). According to Weinstock & Hoppszallern (2015), organizations on the Most Wired list are consistently improving their patient engagement by connecting daily with patients through the Internet, such as providing education and allowing for e-visits with the health care team. Additionally, the Most Wired organizations note the importance of patient portals being user-friendly and useful (Weinstock & Hoppszallern, 2015). By editing the TRT, researchers felt the addition of patient engagement better reflected today's technology.

Limitations

Limitations to this study include those that exist within the HLEHHC instrument. Options for responses on each of the tools within the HLEHHC manual are limited to a 3-point scale. The preferred scale is a 5- or 7-point scale, which results in data being available on a continuum from strongly agree to strongly disagree; therefore, offering a richer data-set. In addition, the method of using a website and personal knowledge as an authoritative source for the assessment of technology was a novel approach, and outdated items on the technology form were updated by the researchers; therefore, challenging content validity.

The cross-sectional design of the print assessment limits the ability of researchers to infer “causal effect” due to lack of randomization. We cannot say with certainty that results found with the sample of 150 documents we reviewed would be duplicated in the whole population documents. Future research should include a truly randomized sample of the total number of documents.

The low inter-rater reliability between raters using the PEMAT precluded using the data from that part of the print assessment study; therefore, we did not have valid data on the “actionability and usability.” Further research should be done to understand why there was a low inter-rater reliability and to further explore the validity of this tool.

Conclusion

Health literacy affects people of all ages and education levels. The National Action Plan to Improve Health Literacy calls for a focus on systematic problems rather than potential shortcomings of patients (U.S. Department of Health and Human Services, 2010). At UTMC, a librarian-led task force was created to assess the organization's current state of health literacy and to serve as a catalyst for promoting changes at UTMC. The HLEHHC offered a set of tools to measure aspects of the health care organization that impact patients with low health literacy. As previously mentioned, the HLEHHC was used at UTMC to assess the health literacy environment of the medical center; included in this report were Print Communication and Technology.

HLEHHC was a useful way to evaluate an organization's health literacy attributes in relation to print and technology. Health care organizations that do this demonstrate commitment to patient-centered care. UTMC's score for both Print Communication and Technology ranked in the highest of the three-category scoring rubric, which translates within the HLEHHC scoring rubric as “continue to monitor and eliminate literacy-related barriers.” Researchers took a unique approach to reviewing and reporting the data for each tool on a question-by-question basis; therefore, revealing more granular, actionable information on where there are opportunities to improve the health care environment for all patients. This analysis resulted in proposing specific actions based on best practices that UTMC could implement in the coming year.

Future plans for UTMC in regards to print communication include the following: the task force members will provide instruction to medical center team members on how to create easy-to-read and engaging patient education; create an advisory committee to evaluate the cultural sensitivity of the print communication; implement focus groups to evaluate print communication; and research vendors to find one that offers patient education written below the sixth-grade level. Future plans regarding technology include providing an engaging patient portal and promoting the use of a smart phone app for accessing patient portals.

Table A

Technology Rating Tool Revisions

Question Number	Original	Revision
2	“Televisions are equipped for DVD or VCR use.”	“Televisions can deliver digital health information videos.”
9	“Computers are programmed for orientation purposes.”	“Computers have capacity for education purposes.”
10	“Computers are programmed for educational purposes.”	“Computers have Internet connection.”
11	“Computers have Internet connections.”	“Computers have access to social media.”
12	“Computers have headsets connected to them.”	“Computers have headsets available to be checked out.”
15	“Kiosks are available to patients in one or more locations (i.e., waiting areas, testing sites, pharmacy, resource rooms).”	“Patients can access their test results online.”
16	“Kiosks are programmed for orientation purposes.”	“Patients can access their prescription history online (i.e., patient portal).”
17	“Kiosks are programmed for educational purposes.”	“Patients can request health information from their room.”
18	“Kiosks have headsets connected to them.”	“Patients can request ‘video chat’ from their rooms.”

Table B

Technology Rating Tool Rankings

Statement	Section	Rank
Televisions are available to patients in one or more locations.	Televisions	3
Televisions can deliver digital health information videos.	Televisions	3
Televisions are used for orientation purposes.	Televisions	3
Televisions are used for educational purposes.	Televisions	3
House telephones are available to patients in one or more locations.	Telephones	3
Computers are available to patients in one or more locations.	Computers	3
Computers have capacity for educational purposes.	Computers	3
Computers have Internet connection.	Computers	3
Computers have access to social media.	Computers	3
Patients can access their test results online.	Patient engagement	3
Patients can access their prescription history online.	Patient engagement	3
Patients can request health information from their room.	Patient engagement	3
Patients can request “video chat” from their rooms.	Patient engagement	3
Computers have headsets available to be checked out.	Computers	2
Examination rooms have computers to show patients their medical record.	Computers	2
Providers can print out specific patient education.	Computers	2
House telephones offer direction to people throughout the facility.	Telephones	1
House telephones offer links to translation services.	Telephones	1