Assessing the understandability, actionability, and quality of online resources for the self-management of bipolar disorder.

INTRODUCTION: This study aims to assess the understandability, actionability, and quality of online resources for the self-management (SM) of bipolar spectrum disorders in adults.
METHODS: An online search using Google, Bing, and Yahoo! search engines was conducted to identify resources for bipolar disorder. Those that were published in English, discussed at least 1 method directed at improving an SM task, and were within the first 25 nonadvertisement results for each search were included. Resources directed specifically at adolescents were excluded. Understandability and actionability of the online resources were evaluated using the Patient Education Materials Assessment Tool (PEMAT). Quality of the online resources was evaluated using the DISCERN instrument. The number of SM tasks each resource discussed was also evaluated. Overall mean appropriateness was calculated by averaging the percentage scores of understandability, actionability, and quality.
RESULTS: Fifty-two resources were included. The mean sample scores were 8.4 (SD, 2.1; range, 2-13; maximum, 15) for understandability, 2.2 (SD, 1.2; range, 0-4; maximum, 5) for actionability, and 46.1 (SD, 8.9; range, 30-57; maximum, 75) for quality. The overall mean appropriateness percentage was 53.5% (SD, 11.7%; range, 18%-77%), with a goal of at least 70%. Included resources addressed a mean of 7.1 tasks (SD, 2.5; range, 1-14; maximum, 20). DISCUSSION: Most online resources for the SM of bipolar disorder scored poorly for understandability and actionability based on PEMAT scores and had low to moderate scores for quality using the DISCERN instrument. Future online resources should be designed with the goal of increasing appropriateness for patients.

Introduction

According to the National Institute of Mental Health,1 approximately 7.2 million American adults (2.8%) are affected by bipolar disorder. Patients with bipolar disorder may have significant impairment physically, emotionally, socially, and in occupational functioning. Because of the nature of bipolar disorder, undertreated or untreated individuals typically have relapses, significant symptoms, and severe impairment.1 Up to 20% of individuals with bipolar disorder, most of whom are untreated, die by suicide.2,3 Approximately half of individuals with bipolar disorder are untreated in any given year.4 Common barriers to seeking and receiving effective treatment include social stigma, limited knowledge about mental illness, lack of perceived need for treatment, and health care system–level barriers, such as availability of providers or cost of treatment.5

Self-management (SM) interventions attempt to minimize barriers to treatment, reducing the risk of relapse.6 Broadly, SM interventions empower patients with the information and skills necessary for the successful day-to-day management of their chronic condition when they are between visits with providers, therapists, and other health professionals. The interventions are intended to complement and be implemented in conjunction with more traditional treatment modalities, such as pharmacotherapy, psychotherapy, and lifestyle modifications. SM activities can be categorized into 3 types of tasks: focusing on illness needs, activating resources, and living with a chronic illness.7 Focusing on illness needs outlines the tasks and skills necessary for individuals to take care of their bodies and illness-specific concerns, such as taking all medications as prescribed and recognition of early symptoms of a mood episode.7 Activating resources covers maximizing support from individuals (eg, family members, friends, health care providers) and community resources and services (eg, spiritual resources, social and transportation services) to aid in optimal disease management.7 Lastly, living with a chronic illness covers the tasks and skills related to coping with the illness and growing as a person; processing emotions, adjusting to illness and new self, and integrating into daily life are potential examples.7 The benefits of implementing tasks within these 3 core SM areas include promoting active engagement of patients in their treatment and prevention of relapse; improved confidence with medical management, role management, and emotional management; and increased self-sufficiency and decreased patient reliance on health care professionals. These benefits culminate in improved quality of life, clinical outcomes, and functional outcomes for individuals with chronic illness.6-8 Therefore, SM interventions can and should be recommended to any patient with a chronic disease willing to implement them, regardless of disease state or severity.

Although SM resources are commonly given to patients by health care providers, the Internet represents a secondary source of readily accessible SM resources. The first step for most people seeking specific health information is to use search engines.9 However, the appropriateness, quality, and consistency of online health-related search results is questionable. Zhang et al10 found the overall quality of online health information was problematic and that online materials contained outdated or even harmful information. Rathod et al11 determined online resources for the SM of depression averaged a grade 10 reading level, exceeding both the ideal grade 5 reading level and the acceptable grade 8 reading level. Considering the potential benefits of implementing SM tasks in bipolar disorder and the significant number of patients seeking health information via online search engines, there is value in evaluating the appropriateness of online resources found by Internet searches for the SM of bipolar disorder.

The aim of this study is to assess the understandability, actionability, and quality of these online resources to determine their overall appropriateness. Understandability refers to when consumers of different backgrounds and varying levels of health literacy can process and explain key information.12 Actionability refers to the ability of these consumers to identify what they can do based on the information presented.12 Quality refers to resources being created using good evidence and aligning with current treatment guidelines and evidence-based practices.13

Methods

This study identified online resources for the SM of bipolar disorder through an online search conducted in February 2020. Google, Bing, and Yahoo! search engines were used to identify up to 78 online resources—up to 2 from each completed search. A total of 13 search terms (Table 1) were used within each of the 3 search engines for a total of 39 completed searches. The search terms were modeled after Rathod et al,11 a similar study for online SM resources for depression which used search frequency data to determine search terms most likely to be used by patients. The goal of the study was to find resources patients would likely locate in their own searches. As such, the inclusion criteria for the study were lenient. Resources were eligible for inclusion if they discussed at least 1 method directed at improving a SM task (Table 2) and were published in English. Resources also had to be within the first 25 nonadvertisement search results for each search conducted, which mimicked typical search behavior of prioritizing early results and provided a defined cutoff to terminate the search when repeated results increased in prevalence. The first 2 nonadvertisement, nonrepeated resources for each search were included. Resources directed at adolescents with bipolar disorder were excluded. The primary researcher evaluated all identified resources to determine if they met inclusion criteria. The search was conducted from a central US Internet protocol (IP) address using the incognito mode of the Google Chrome Web browser. The browser history, cache, and cookies were cleared prior to the search to mitigate past searches imparting bias on the study search results.

TABLE 1

Used search terms

Search Terms
Fixing bipolar disorder	Bipolar disorder self- management
Managing bipolar disorder	Bipolar disorder self-care
Dealing with bipolar disorder	Bipolar disorder guide
Living with bipolar disorder	Bipolar disorder education
Coping with bipolar disorder	Bipolar disorder patient education
Handling bipolar disorder	Bipolar disorder toolkit
Bipolar disorder self-help

TABLE 2

Incidence of self-management tasks (n = 52)

Self-Management Task	Components	No. Incidence (%)
Focusing on illness needs
Learning about condition and health needs	Learning information, skills, and strategies	28 (53.8)
Recognizing and managing body responses	Monitoring/managing symptoms, side effects, and body responses Recognizing limits	45 (86.5)
Completing health tasks	Keeping appointments Managing/taking medications	34 (65.4)
Becoming an expert	Goal setting, decision-making, and problem-solving Developing confidence and self-sufficiency	6 (11.5)
Changing behaviors to minimize disease impact	Modifying diet, nutrition, and physical activity Reducing stress	46 (88.5)
Sustained health promotion activities	Maintaining screenings and lifestyle modifications Using complementary therapies	29 (55.8)
Activating resources
Creating/maintaining relationships with providers	Communicating effectively Making decisions collaboratively	27 (51.9)
Navigating the health care system	Coordinating services/appointments, insurance Using resources effectively	15 (28.8)
Benefiting from psychologic resources	Drawing on intrinsic resources (eg, creativity) Maintaining positive outlook, hope, and self-worth	14 (26.9)
Sustaining spiritual self	Being part of a spiritual community, praying	5 (9.6)
Obtaining and managing social support	Seeking support of family and friends Being part of a community of peers with similar experiences	34 (65.4)
Addressing social and environmental challenges	Seeking resources, such as financial assistance, environmental support, and community resources	1 (1.9)
Living with a chronic illness
Processing and sharing emotions	Exploring and expressing emotional responses Dealing with shock of diagnosis, self-blame, and guilt	5 (9.6)
Adjusting to illness	Developing coping strategies Dealing with discouraging setbacks	23 (44.2)
Adjusting to new self	Dealing with stigma	3 (5.8)
Modifying lifestyle to adapt to disease	Creating a consistent health routine Controlling environment	23 (44.2)
Seeking normalcy in life	Managing disruptions in school, work, family, and social activities Balancing living life with health needs	20 (38.5)
Reevaluating life	Reframing expectations of life and self Coming to terms with terminal condition and end of life	0 (0)
Personal growth	Learning personal strengths and limitations Becoming empowered and being altruistic	7 (13.5)
Striving for personal satisfaction	Finding meaning in work, relationships, activities, and spirituality Creating a sense of purpose and appreciating life	6 (11.5)

All included online SM resources were evaluated by the primary researcher using the Patient Education Materials Assessment Tool (PEMAT), a validated instrument that measures both the understandability and the actionability of patient education resources.12 The PEMAT is available in 2 forms, PEMAT-Print and PEMAT-Audiovisual. Only the PEMAT-Print, which includes materials that can be printed or viewed from a Web site, was used for the evaluation of included resources. No formal training is required to use the PEMAT, although a detailed user's guide is available. The PEMAT includes a total of 19 questions for understandability and 7 questions for actionability.12 Answer options include Agree (1 point), Disagree (0 points), or Not Applicable (excluded when calculating total score, decreasing the total possible points).12 Four items in the PEMAT: Understandability domain were not included in the scoring because of a lack of presence in the resources: clear and easy to understand numbers, no required calculations, easy to hear audio, and simple tables with short and clear row and column headings. Likewise, 2 items in the PEMAT: Actionability domain were not included in the scoring: simple instructions or examples of how to perform calculations and explains how to use figures. Therefore, the PEMAT: Understandability domain scores were calculated out of 15 total possible points and the PEMAT: Actionability domain scores were calculated out of 5 total possible points. Both scores were summed, and a percentage was calculated. A resource was considered understandable and/or actionable if it scored ≥70% on either section, which was used as the goal result for the PEMAT in previous studies.11-13

All included SM resources were also evaluated by the primary researcher using the DISCERN instrument, a validated tool that assesses the quality of patient education resources.14 No formal training is required to use the DISCERN instrument, although detailed guidance is available regarding the scoring of each question. The DISCERN instrument includes 16 items that assess the quality of materials in 3 sections. The first 8 items assess the reliability of the publication and trustworthiness of information. The next 7 items assess specific details of the information concerning treatment choices. The final item is an overall rating of the material and was omitted from this study to allow a more detailed breakdown of quality scores. Each item is rated from 1 to 5, with higher scores reflecting higher overall quality, for a maximum possible score of 75. Scores on items 1 to 15 were summed and a percentage calculated based on the total possible of 75. A resource was considered to be high quality with a score of ≥70%, which was used as the goal result for the DISCERN instrument in a previous study.11 Lastly, an overall mean appropriateness percentage for each resource was calculated by averaging the percentage scores of the 3 scales. This calculation was developed to give an assessment of the resource as a whole. Previous studies did not calculate composite scores for resources.

An SM task checklist consisting of 20 individual tasks split between 3 core areas (Table 2) was developed based on research by Schulman-Green et al.7 All resources were evaluated based on the number of SM tasks included. Other information, including publication location, publishing year, last updated year, publication entity (eg, private organization vs nonprofit entities), format (eg, Web page vs electronic booklet), and presence of advertisements, was also collected. All study data were collected and managed using REDCap electronic data capture tools hosted at the Center for Health Insights of the University of Missouri-Kansas City.15 Following data collection, data were exported to Microsoft Excel in order to calculate descriptive statistics for the 2 PEMAT domains and the DISCERN instrument.

Results

The Internet search identified 52 resources (Table 3),16-67 with resources published from the United States (n = 39), United Kingdom (n = 5), Canada (n = 4), and Australia (n = 4). Only 52 resources were eligible for inclusion as repeat search results increased in prevalence, causing some searches to terminate after 25 nonadvertisement results with fewer than 2 resources identified. Most resources were created by private organizations (n = 31), with the remainder created by professional organizations (n = 3), universities (n = 1), hospital systems (n = 2), government organizations (n = 2), or other nonprofit entities (n = 13). Most of the resources were Web pages (n = 42). The rest were formatted as personal blogs (n = 7) or electronic booklets (n = 3). Approximately 30% of resources were missing a year of publication or a last updated year, and 6 resources were missing both. Most resources contained advertisements (n = 31) directed at readers.

TABLE 3

Characteristics of top and bottom 10 results included in final sample

No.	Title	No. of SM Tasks	PEMAT		DISCERN Score, %	Overall Approp, %
			Und Score, %	Act Score, %
1	Living with bipolar disorder16	11	86.7	80	72	79.6
2	Self-care for managing mania17	8	73.3	80	62.7	72
3	Self-help strategies for bipolar disorder: Lifestyle changes and crisis plans18	8	60	80	73.3	71.1
4	Bipolar disorder–information for patients and families19	9	80	60	70.7	70.2
5	How to cope with bipolar disorder (manic depression)20	9	80	60	66.7	68.9
6	Bipolar disorder21	8	73.3	60	70.7	68
7	What to know about bipolar disorder and anger22	9	80	40	77.3	65.8
8	Bipolar disorder23	6	53.3	60	82.7	65.3
9	Self-care for bipolar disorder24	7	60	80	50.7	63.6
10	Self-management techniques for bipolar disorder25	3	53.3	80	57.3	63.5
43	50 Natural ways to manage and overcome symptoms of bipolar disorder26	14	26.7	40	58.7	41.8
44	Bipolar depression management tips27	9	66.7	0	57.3	41.3
45	Tips to successfully manage your bipolar disorder28	5	40	20	61.3	40.4
46	How to practice self-care when you have bipolar disorder29	9	46.7	20	50.7	39.1
47	Living with bipolar disorder30	4	46.7	20	50.7	39.1
48	Depression in bipolar disorder: What you can do31	6	26.7	40	45.3	37.3
49	Self-help for bipolar disorder32	4	40	20	50.7	36.9
50	Psychotherapy and self-help for bipolar disorder33	5	46.7	0	57.3	34.7
51	My story: Bipolar disorder and self-care34	7	53.3	0	42.7	32
52	Managing bipolar disorder35	1	13.3	0	40	17.8

Act score = actionability domain score; Approp = appropriateness; PEMAT = Patient Education Materials Assessment Tool; Und score = understandability domain score.

The mean PEMAT: Understandability score was 8.4 (SD, 2.1; range, 2-13) out of a possible 15 points. Seven resources (13.5%) in the understandability domain included relevant visual aids and 13 resources (25%) included a summary of key points. The mean PEMAT: Actionability score was 2.2 (SD, 1.2; range, 0-4) out of a possible 5 points. Within the actionability domain, 21 resources (40.4%) broke at least 1 action down into manageable steps, and 12 resources (23.1%) included a tangible tool, such as a mood charting tool, to aid in completion of an action. The mean DISCERN instrument score assessing quality was 46.1 (SD, 8.9; range, 30-57) out of a possible 75 points. One quality concern noted was poor referencing of source materials. A full reference list was available for 10 resources (19.2%), in-text citations alone were used in 15 resources (28.8%), and no referencing information was found in 29 resources (55.7%). Additionally, 16 resources (30.8%) provided external sources of information beyond the publishing Web site. The overall mean appropriateness percentage was 53.5% (SD, 11.7%; range, 18%-77%). The Figure displays the appropriateness percentage for each of the 52 included resources. Four resources (7.7%) had an appropriateness score reaching or exceeding 70%. Of the 20 SM tasks, the reviewed resources addressed a mean of 7.1 tasks (SD, 2.5; range, 1-14). The most commonly included SM tasks were changing behaviors to minimize disease impact (n = 46), recognizing and managing body responses (n = 45), and obtaining and managing social support (n = 35). The least common were reevaluating life (n = 0), addressing social and environmental challenges (n = 1), and adjusting to new self (n = 3). The incidence of the other SM tasks is detailed in Table 2.

FIGURE

Overall percent appropriateness for included resources

Discussion

Despite the potential benefits of implementing SM tasks in bipolar disorder and the prevalence of patients seeking health information via online search engines, this is the first study to evaluate the appropriateness of online SM resources for bipolar disorder. This study demonstrated that most easily accessible online SM resources for bipolar disorder scored poorly on understandability and actionability domains.

Understandability of these resources could be improved with consistent inclusion of short summaries of key takeaway points and relevant visual aids. Actionability of these resources could be improved if resources more consistently broke actions down into explicit, manageable steps and provided tangible tools such as mood charting templates.

Although DISCERN instrument quality scores were consistently the highest scoring aspect, many resources had characteristics indicative of low-quality resources. These included a low rate of referencing sources of information and a high rate of only providing additional sources of information within the hosting site. These deficiencies could easily be remedied by sourcing information used with in-text citations and by providing a variety of outside organizations and Web sites for patients seeking additional information.

This review differs from similar studies because of the decision to not include readability assessments. The applicability of readability assessments to educational materials is limited because many of the tools score readability based on either the number of letters or syllables of words in a sample of the publication. Educational materials for patients often require the repetition of certain long or complex words that are potentially vital for complete education. For example, in bipolar disorder materials, it is not uncommon to see words such as depression, hypomania, stabilization, and anticonvulsant. When appropriately explained, such terms can elevate educational resources rather than limit them. Additionally, similar studies have found a lack of correlation between readability levels and understandability scores as assessed by the PEMAT.11,13

This study has several strengths, including a reasonably large sample size of resources (n = 52). Additionally, resources were identified using online search strategies designed to mirror those used by consumers, and the inclusion criteria were lenient. Lastly, validated tools such as the PEMAT and the DISCERN instrument were used to evaluate the resources. This study also has limitations. Using search engines to identify resources creates a bias toward popular sources of information rather than reputable or high-quality sources of information. Additionally, the search was conducted from the central United States; therefore, search results studied may be different from results from other areas of the country or world. Although both the PEMAT and the DISCERN instrument include in-depth scoring rubrics, many scoring items relied on subjective interpretation. Lastly, although the DISCERN instrument addresses sources of information, the accuracy of the information included in each resource was not evaluated directly.

Conclusion

Because of low scores for understandability and actionability and low to moderate scores for quality, the average overall appropriateness score of online resources for the self-management of bipolar disorder was below the goal of at least 70%. Online resources should be updated or designed with the goal of increasing understandability, actionability, and quality. The PEMAT domains and the DISCERN instrument questions could be used as a guide when designing these resources.