Performing an Organizational Health Literacy Assessment in a Shelter Serving People with Mental Illness.

BACKGROUND: Health literacy research and practice are constantly evolving. Recent inquiry has highlighted the burdensome literacy demands faced by people with serious mental illness (SMI). Systems, organizational operations, and structures can play a role in decreasing literacy demand, thereby reducing unnecessary challenges for people with SMI. Brief Description of Activity: An organizational health literacy assessment was conducted to explore literacy demands in one mental health shelter and revealed best practice action steps for improving the literacy environment. IMPLEMENTATION: The assessment included an exploration of the shelter environment using The Health Literacy Environment Activity Packet, First Impressions & Walking Interview, and a commonly used shelter document using the Simple Measure of Gobbledygook (SMOG), Suitability Assessment of Materials (SAM), and Centers for Disease Control and Prevention (CDC) Index.
RESULTS: The literacy demands of the shelter environment and a frequently used document exceeded the literacy skills of people with SMI. Environment assessment revealed environmental facilitators (e.g., welcoming atmosphere) and barriers (e.g., unclear signage). Document assessment also revealed facilitating factors and barriers. SMOG scores ranged from 11.25 to 11.80 (median: 11.38), meaning 11th to 12th grade-level reading skills are required to understand, use, and take action on the document's content. A SAM score of 50% (adequate) and a CDC Index score of 42.1 (revise and improve) reveal materials contain both facilitating factors (e.g., chunked sections) and barriers (e.g., jargon, mismatched graphics) to use. LESSONS LEARNED: The mismatch between system demands and the literacy skills of people with SMI is more profound than that of the general United States population. Organizational health literacy assessments are achievable and useful for both immediate and long-term action aimed at understanding and improving the organizational health literacy of mental health spaces. Further work is needed to explore the role of behavioral health services in addressing the institutional and programmatic literacy demands that inhibit treatment and recovery. [HLRP: Health Literacy Research and Practice. 2022;6(3):e167-e174.] Plain Language Summary: An organizational health literacy assessment reveals how system demands can be changed to better serve intended users. Engaging in mental health, recovery, and treatment services requires complex literacy skills. Generally, the U.S. adult population does not have the skills to meet such demands, and this is especially true among public mental health service users.

Growing attention has been given to the role of literacy in health and health care. However, less work has examined the role of literacy in the lives of people with serious mental illness (SMI) and their experiences of mental health, especially in mental health care systems. This includes health care access and participation in treatment, recovery, and community life. Better understanding how literacy affects the lives and health care of people with SMI is important for reducing health inequities and supporting meaningful recovery and engagement (Arnott et al., 2011; Christensen & Grace, 1999; Gazmararian et al., 2000; Hayes & O'Grady, 2003; Lincoln et al., 2006; Lincoln et al., 2008; Lincoln et al., 2015; Sentell & Shumway, 2003; Weiss et al., 2006).

Descriptions of the relationship between health and individual literacy skills are most prevalent in the literature, suggesting that higher individual literacy skills are related to more positive health outcomes (Berkman et al., 2011; Nielsen-Bohlman et al., 2004; Office of Disease Prevention and Health Promotion, 2010). Yet, an increasing evidence base also demonstrates that environments (e.g., institutional or organizational settings and related systems) can reduce health literacy demands to ease the burden of understanding and performing health-related tasks or actions. These can include changes to materials, interactions, websites, signage, processes, and spaces in clinics, hospitals, and beyond (Ownby et al., 2012; Rosenfeld et al., 2013; Rudd, 2017; Sudore & Schillinger, 2009).

Healthy People 2030 defines organizational health literacy as “the degree to which organizations equitably enable individuals to find, understand, and use information and services to inform health-related decisions and actions for themselves and others” (Office of Disease Prevention and Health Promotion, 2021). This new definition along with prior research highlights the importance of organizational health literacy in achieving optimal health (Brach et al., 2012; Groene & Rudd, 2011; Horowitz et al., 2014; Rudd, 2017). Mental illness and limited literacy are both highly stigmatized statuses (Lincoln et al., 2017). As such, disclosure of reading or other literacy difficulties among people using public mental health services is even more complex and challenging, suggesting that system-level changes could have important benefits.

To explore organizational health literacy in the mental health setting, a team was developed (The Literacy Interventions Team, hereafter The Team) including partners who had recently been part of a study to explore the prevalence of limited literacy in the lives of people living with mental illness using public mental health outpatient services. Study participants had lower rates of literacy than the general U.S. adult population, with accompanying challenges in navigating mental health systems (Lincoln et al., 2021). The Team, including The Massachusetts Department of Mental Health (MDMH) (one of the study partners), developed a three-part strategy to be implemented at interested study sites: (1) develop and conduct reading skills groups, (2) create a drop-in program to help clients fill out forms and read/organize (e)mail, a need specifically identified by study participants, and (3) assess the organizational health literacy demands of a mental health setting (Lincoln et al., 2021).

To our knowledge, there has been no exploration of organizational health literacy in the context of mental health care settings, systems, and structures (Berkman et al., 2011; Martin et al., 2011; National Center for Education, 2011; Office of Disease Prevention and Health Promotion, 2010; Rampey et al., 2016; Schapira et al., 2009). Here, we describe a pilot organizational heath literacy assessment in a shelter serving people with mental illness, part 3 of the strategy outlined above.

Brief Description of Activity

MDMH asked The Team to conduct the pilot in one of their shelters for people with mental illness. The chosen shelter provides transitional housing services to nearly 50 homeless people who have severe and persistent mental illness and who are actively engaged in treatment geared to helping residents obtain and retain permanent housing. All residents have a treatment/service plan and may also attend other outpatient services (BWH Bulletin, 2021). Noting the high incidence of nonadherence to shelter community rules, MDMH and shelter leadership were curious whether residents understood the “house rules,” especially because serious infractions (e.g., persistent substance use, violence) may lead to suspensions and ultimately expulsion. Therefore, a goal of this pilot was to understand shelter facilitators and barriers for remaining housed, since stable housing is linked to better mental health opportunities and care (Reif, 2014).

Implementation

An organizational health literacy assessment explores whether a program, organization, or institution is equitably enabling individuals to find, understand, and use information and services (Office of Disease Prevention and Health Promotion, 2021). It identifies literacy-related facilitators and barriers and related change ideas to improve navigation for everyone. (Groene & Rudd, 2011; Horowitz et al., 2014; Rudd & Anderson, 2006; Rudd, 2017). For this pilot, we (1) assessed navigation at the shelter itself, (2) determined appropriateness of the most commonly used written document, and (3) provided results and practical recommendations to shelter leadership and patient stakeholders. The entire assessment process including the report was conducted over a period of 2 months in 2018 at the request of MDMH.

Assessment Tools

We used The Health Literacy Environment Activity Packet, First Impressions & Walking Interview to assess the shelter environment (Rudd, 2010). It is a low-time burden, publicly available overall snapshot of organizational health literacy designed for two participants to conduct. One person acts as Guide to facilitate the process (person more familiar with the institution) and the other as Observer to share impressions (person less, or not, familiar with the institution). Some advanced planning is required. The first step is First Impressions. To complete this section, the Observer uses a starting location predetermined by the Guide, based on where patients/participants commonly live, or might start out from. Step-by-step questions lead the Observer through three structured activities: (1) Telephone: calling the institution, attempt to get directions; (2) Website: visiting the institution website, attempt to get directions, and (3) Walk to the Entrance: walking to the institution entrance, observing facilitators and barriers. Examples of questions for each activity include observing the live voice or automated answering system, determining the availability of website directions and burden of locating them, and noting the frequency and helpfulness of signage and staff members. There are also open-ended prompts for describing impressions and identifying barriers and facilitating factors.

The second step is The Walking Interview. The Guide invites the Observer to imagine they are visiting the organization for the first time and to take a thoughtful look around. The Guide then prompts the Observer to share thoughts and make observations using step-by-step suggestions, as they make their way to a public destination predetermined by the Guide. There are six stages: (1) Observations at the Entry Point or Lobby (What is the overall literacy environment?); (2) Directions/Seeking Help (Is help with navigation available?); (3) Navigation (What is it like to navigate to a specific location?); (4) Observation (What are the literacy demands or assumptions patients encounter as they access services?); (5) Reflections (What is the overall impression [e.g., use of written word, navigation aides, signs, language]?); and (6) Feedback (What was learned? What are next steps?). Each stage includes detailed follow-up questions by the Guide to highlight facilitators and barriers of specific related aspects. Emerging themes are generated from the detailed observations and impressions of the observer.

In this pilot, two components of the tool could not be completed for the shelter because there is no website or phone number. Access and location are gained via referral from other state programs. Therefore, only the “Walk to the Entrance” of First Impressions and The Walking Interview were completed. Further details were obtained about the shelter's purpose and perspective through a conversation with the Director.

The shelter's guidelines and living contract is provided to shelter guests as one packet of required reading. It must be signed before becoming part of the shelter community and all its rules must be followed to remain a guest. It includes a cover letter, “guidelines for guests living at [shelter name redacted],” “living contract,” and signature pages. It contains information about the daily schedule, living rules, and consequences for breaking rules. The signature sheets ask guests to sign before their stay is approved, which means they acknowledge reading and understanding the living contract. The signature pages were excluded in the review because there was almost no text to assess.

To evaluate materials, we used three assessments: the Simple Measure of Gobbledygook (SMOG) (Laughlin, 1969; Scott, 2003), the Suitability Assessment of Materials (SAM) (Doak, 1996), and the Centers for Disease Control and Prevention Clear Communication Index (CDC CCI) (CDC, 2019). These assessments were chosen because they are publicly available online, provide a wide breadth of information health literacy experts consider important, and include concrete guidance for improving information (Baur & Prue, 2014; O'Sullivan et al., 2020). The SMOG calculates a score which describes the grade-level skills needed to engage with the text. Step-by-step calculation instructions consider total polysyllabic words and sentence length. The SAM assesses the suitability of print, digital, video or audio content. There are six areas: content, literacy demand, graphics, layout, and typography, learning stimulation/motivation, and cultural appropriateness. Items in each area are assigned a 0, 1, or 2, based on descriptions provided in the tool and directions. An overall percentage score falls into one of three categories: superior, adequate, or not suitable, calculated based on 44 possible points. The CDC CCI provides a score based on 20 items across four parts. Each item is assigned a 0 or 1 based on descriptions provided in the tool and directions. The Index has four sections: (1) Core (e.g., audience, main message or call to action, language used, information design, state of the science), (2) Behavioral Recommendations, (3) Numbers, and (4) Risk. The score describes the clarity of communication products. A 90 or higher suggests most scored items make the material easier to understand, whereas an 89 or below requires serious reconsideration item-by-item. Two trained Team members [L.R., S.G.] conducted independent assessments of the documents using the SMOG, SAM, and CDC CCI. Then they met to compare scores, discuss discrepancies, and reach consensus. There was no discordance between assessors for the SMOG or CDC CCI. Only two items on the SAM were initially discordant between reviewers, by 1 point each. After brief discussion, all items aligned, and full consensus was reached.

Results

The organizational health literacy assessment of an MDMH shelter revealed facilitators and barriers for guests and staff.

Exploring the Shelter Environment

The shelter's shared kitchen was the predetermined destination for the Walking Interview. The following facilitators and barriers to navigation were revealed. Facilitators: Many rooms (e.g., offices, bathrooms) are labeled with signs in English and Spanish. Both bathrooms and alarmed doors have clear signs with appropriate illustrations and icons. Guest common areas, such as the community room and dining area, are welcoming and comfortable and include access to the internet. Likewise, the kitchen/dining area displays art made by shelter guests. Barriers: The main shelter entrance lacks a visible sign for passersby. Neither the shelter's name nor street address are posted. Many shelter rooms are missing labels. “No smoking” signs are not present in the shelter. Posted signs are not at appropriate readability or comprehensibility levels. Analogue clocks are used throughout the shelter. “Rules reminder” signs were not posted in bedrooms or public spaces to summarize the salient “contract rules” for daily living.

Exploring the Shelter's Guidelines and Living Contract

We divided the document into three sections, per SMOG instructions for long documents. SMOG scores ranged from 11.25-11.80 (median: 11.38), suggesting 11th to 12th grade-level reading skills are required to appropriately understand, use, and take action on the document's content. The opening letter scored 11.8, the Guidelines section scored 11.38, and the Contract section scored 11.25. The SAM score was 50% (17 of 34), which suggests that the document is adequate in its efforts to provide clear and actionable information. The CDC Index score was 42.1, which suggests the document needs to be revised and improved.

The document contextualized information, using headers to guide readers, judicious use of basic numbers, and specific recommendations. New information is typically put into context, aiding reader comprehension. Headers or subtitles are also used frequently, helping readers to anticipate upcoming information or guide them through long content. Numbers are used infrequently which helps alleviate common difficulties with numeracy. When recommendations are specific, they are most clear (i.e., if you are going to smoke, please go outside—far from the shelter entrance).

The documents had a noticeable mismatch between demands (11th-12th grade-level reading skills) and shelter guests' reading skills (about 5th grade) (Lincoln et al., 2021). Also problematic are the lack of brevity, plain language, active voice, and appropriate graphics and formatting. Non-essential information is also included throughout the document, retaining a wide scope of focus. Further, recommendations are generally unclear: written in a passive tone without relevant examples. Likewise, the document's purpose is not clearly laid out at the beginning. Key message boxes or similar are also not present to highlight the most relevant information. Documents had embedded language, long sentences, and jargon. Documents also did not take advantage of large font sizes, bold text, or judicious use of color to emphasize key points. ALL caps and long narrative lists without bullets were also present.

Lessons Learned

Overall, we note the following best practice recommendations and lessons learned. Most are broadly applicable to health and health-related organizations as practical starting points for improving understandability, usability, and actionability of materials and environments. This is never a punitive process. We focus on addressing barriers, but also on enhancing current facilitators, an essential part of achieving optimal organizational health literacy. In the case of this pilot, there is a long-standing collaboration, which has allowed open dialogue regarding strategic next steps at the shelter. We have also discussed systems change more generally. We are ready to support implementation at the shelter and other MDMH settings, as appropriate and requested.

Exploring each assessment item-by-item reveals opportunities for improvement. Shelter staff could create or revise written materials for easier use; this includes a reading level of 5th grade or below (Lincoln et al., 2021). Staff could also break up long documents into multiple shorter documents or more clearly delineate sections to increase the focus and clarity of each. Nonessential information could also be eliminated to narrow scope and make documents more accessible. Any recommendations provided in materials should be direct, using specific examples to improve clarity. For example, in the Guidelines & Living Contract, the statement “Physical violence or personal harassment of any kind will not be tolerated” could be followed with “Do not hit, kick, yell at, or touch any other guest or staff member at the shelter.” Furthermore, the purpose of each document should be clear on the opening page; a key message box can be used to highlight the most relevant information. Larger font sizes, bold type, or judicious use of color can be used to emphasize key points. Finally, staff currently give the Guidelines & Living Contract to new guests to read on their own as part of the shelter orientation. Instead, staff could engage new guests in an interactive discussion of the materials to allow time for clarification and “check-ins” about the information presented, thereby increasing comprehension and hopefully the length of guest stays at the shelter.

Exploring The Health Literacy Environment Activity Packet: First Impressions and Walking Interview item-by-item reveals many shelter-level opportunities to reduce literacy-related barriers to participation. To increase “ease of use” and “ownership” of shelter spaces, additional communal area components could be added, such as resident-made art or spaces for books/magazines/games. Rooms could be labeled in plain language throughout the shelter, in both English and Spanish, to reflect the primary languages spoken by guests. Staff could put “No Smoking” signs in “typically offending” areas (e.g., bathrooms) to remind residents of the no smoking policy. Staff could redesign posted signs in common spaces to be larger, displaying less text and more appropriate font and size formats to improve readability and comprehension. Installing digital clocks in common areas may also reduce burden for guests required to follow the shelter schedule, since time concepts (including telling time) are particularly challenging for people with lower literacy/numeracy skills (French, 2014; Harris, 2008). Managing time is an essential piece of understanding and acting on daily schedule rules that are part of the Living Contract agreement (e.g., attending meetings/appointments, common area hours, curfew). Posting plain language, well-designed “rule reminder” signs in each bedroom could also help prompt residents to observe the daily living rules.

Being a shelter guest requires successfully navigating the shelter environment, which is critical for receiving ongoing mental health treatment and recovery services. Literacy-related barriers can jeopardize treatment “adherence” and the ability to maintain stable housing, both crucial to seeking treatment and participating in recovery.

Reflections from Shelter Leadership and Stakeholders

After the dissemination of The Team report, MDMH shelter leadership reflected on what they learned as well as next action steps. In all, they described the assessment as “eye-opening,” allowing them to understand the shelter through the eyes of their guests, especially the experience of interacting with the shelter for the first time. The shelter Director noted, “It helped me think about the way we conduct business. We get quite verbose in our paperwork because we want to make sure everything is covered, but the reality is we're expecting people to process things at a level that may not be realistic.” This realization informed changes to the written materials clients receive at “check-in,” particularly consolidating long packets of paperwork into shorter, clearer forms, aiding more attainable treatment and recovery goals. It also prompted the posting of important rules in common spaces and every shelter bedroom, improving the expectations and communication between patients and staff.

Leadership noted many other new learnings gleaned from the organizational health literacy assessment, which have not yet been implemented. Execution of planned changes have understandably been slowed by the coronavirus disease 2019 pandemic, which has made even typically challenging implementation more difficult. Plans include plain language updates to client paperwork and signage, including simplifying the language, shortening long documents, adjusting reading levels to meet guest needs, and engaging guests with clarifying questions to make sure they understand the rules and policies of the shelter. MDMH Leadership noted, “We want to create a safer environment for people to express concerns around literacy and make it easy for people to understand expectations, make people feel comfortable disclosing if they don't understand something.” Leadership also perceives organizational health literacy assessments as useful for infusing literacy into conversations and initiatives throughout mental health practice and administration, something they do not feel is regularly considered at present. For this, they highlighted the importance of having a champion who owns the literacy work as a critical part of efficiently and effectively moving it forward.

Notably, the MDMH shelter pilot organizational health literacy assessment has influenced practice throughout the MDMH system, including activities such as a consistent review of the literacy level of documents that are used to inform people of rules or procedures as well as documents used in treatment, such as worksheets. Staff have been trained in ways to present documents (e.g., provide summary of each paragraph, invite questions, ask clarifying questions to assess comprehension) that reduced literacy barriers. Efforts have been made to identify treatment models and evidence-based practices that are designed for people with low literacy including Skills System by Julie Brown (2015). Finally, increased awareness among leadership of the need to address limited literacy in mental health is reflected in other practice changes, such as discussing the prevalence of limited literacy among service users and strategies to improve work with clients in new employee orientations and trainings. Lastly, the MDMH Institutional Review Board changed their policy for consent forms used with people engaging with mental health services. The reading grade level demand of a document must now be below an 8th grade level (Lincoln et al., 2021); previously it was not specified. Likewise, MDMH is making plans to conduct organizational health literacy assessments across MDMH facilities, based on their new understanding of the crucial role of the environment in facilitating access to and receipt of services and treatment in mental health settings. (Rudd, 2017).

Limitations

This assessment was based on best practice methods; nevertheless, limitations must be considered. For example, more expansive assessment efforts were limited by funding and time constraints. Our partners asked us to provide quick feedback (6 months) for a site they considered “high need.” They were in the process of considering institutional changes and wanted the new issues brought to light by the study results to incorporate them into the planning process. Future inquiries should ideally include additional best practices in connecting with stakeholders: (1) focus groups with participants and staff to elicit input on organization materials, policies, and processes, as part of the assessment process; (2) key informant interviews with relevant experts (e.g., mental health, literacy, temporary housing, service users/families); and (3) focus groups with participants and staff to discuss and make decisions about how to implement changes, and evaluate changes over time. Also, more robust environment assessment can be considered (i.e., Health Literacy Environment of Hospitals and Health Centers) (Rudd et al., 2019), Consumer Assessment of Healthcare Providers and Systems (Dyer et al., 2012), Ten Attributes of Health Literate Health Care Organizations (Brach et al., 2012). These limitations do not invalidate reported data, but rather suggest that broader assessment is possible and recommended, where time and funds can be procured.

Implications

Organizational health literacy assessment is an underused strategy in health and health-related organizations. It is a multi-stage endeavor that meets the organization where they are. It highlights opportunities and pitfalls across levels: patients, families, staff, programs, and the broader institution and related systems. Attention to health literacy is a central part of creating health equity, both in process and outcome.

Here, we described an organizational health literacy assessment that identified facilitators and barriers to seeking and engaging in care for people with serious mental illness. Yet, the approach can be applied to myriad other health and health-related settings, where data can be used for improving the accessibility and experience of anyone interacting with the patient and organization, including staff and families. Table highlights recommended steps for conducting an organizational health literacy assessment. Further research is necessary to understand the best practice institutional and programmatic implementation of assessment results for optimal treatment and recovery.

Organizations must examine their role in health, and other, inequities. Implemented alone, person-focused efforts can obfuscate the necessity and possibility of systems change. Institutions and programs generally create higher literacy demands than the average U.S. adult can handle (Berkman et al., 2011; Lloyd et al., 2019; Oelschlegel et al., 2017; Okan, 2019; Rudd, 2017), and for some populations, like people with serious mental illness, the demands are even more profound (Lincoln et al., 2021). Overall, organizational health literacy assessments can help organizations reach many important institutional goals, including reducing costs and improving positive outcomes. Other relevant goals include adherence to medication and other plans, patient engagement, and racial equity. It is a powerful tool for uncovering the mismatch between environment demands and patient skills. Data can be turned into actionable steps for reducing inequitable system burdens, for example via continuous quality improvement. (Brega et al., 2019). U.S. adults are generally at a disadvantage when interacting with health and mental health systems. This is even more true for people with serious mental illness involved in interactions rarely set-up for the pursuit of optimal recovery and treatment. Organizational health literacy assessments can help reveal the systems facilitators and barriers to optimal patient engagement.

Table 1

Highlights: Conducting a Pilot Organizational Health Literacy Assessment

Step 1. Gather the stakeholders. Determine if anyone outside your organization or department is needed Step 2. Do formative work to determine the patient/participant, staff, and leadership concerns and interests related to health literacy (e.g., host informal group conversations) Step 3. Attend publicly available training in health literacy assessment. Invite a speaker. Discuss a key health literacy report or article. Explore the Centers for Disease Control and Prevention (CDC) and Agency for Healthcare Research and Quality (AHRQ) health literacy websites (AHRQ, n.d.; Brega et al., 2015; CDC, 2021) Step 4. Conduct an environment assessment (e.g., The Health Literacy Environment Activity Packet, First Impressions, and Walking Interview) Step 5. Collect 1 to 2 commonly used materials (e.g., instructions, health history form, website) Step 6. Assess the items using rigorous assessments (e.g., Simple Measure of Gobbledygook, Suitability Assessment of Materials, CDC Index). Two people assigned to each material; meet after assessment and come to consensus on findings Step 7. Gather the stakeholders (perhaps a dedicated group now) to explore the results from Steps 4 to 6 Step 8. Determine best next steps with stakeholder group. Consider leadership/organizational policy changes. Use rigorous quality improvements projects within departments Step 9. Repeat steps above. Consider including new information items, additional rigorous assessments (e.g., PMOSE/IKIRSCH (Mosenthal & Kirsch, 1998), numeracy checklist (Apter et al., 2008; Harris, 2008), and more robust organizational health literacy assessments (e.g., The Health Literacy Environment of Hospitals and Health Centers, Version 2) (Rudd et al., 2019)