Comprehension Profile of Patient Education Materials in Endocrine Care.

Introduction: The internet is an ever-evolving resource to improve healthcare literacy among patients. The nature of the internet can make it difficult to condense educational materials in a manner applicable to a worldwide patient audience. Within the realm of endocrinology, there is lack of a comprehensive analysis regarding these pathologies in addition to education materials related to their medical work-up or management. The aim of this study was to assess contemporary online patient education material in endocrinology and management of care.
Methods: Analysis of the readability of 1,500 unique online education materials was performed utilizing seven readability measures: Flesch Reading Ease (FRE), Flesch-Kincaid Grade Level (FKGL), Gunning Fog Index Readability Formula (FOG), Simple Measure of Gobbledygook Index (SMOG), Coleman-Liau Index (CLI), automated readability index (ARI), and Linsear Write Formula (LWF).
Results: The average grade level readability scores from six measures (e.g., FKGL, FOG, SMOG, CLI, ARI, LWF) was more than or equal to 11 which corresponds to a reading level at or above the 11th grade. The average FRE between adrenal, diabetes, and thyroid-related education m aterial ranged between "fairly difficult" to "very difficult". Conclusions: The readability of contemporary online endocrine education material did not meet current readability recommendations for appropriate comprehension of the general audience.

INTRODUCTION

The internet remains a primary source for self-education among patients regarding health-related content. Moreover, the literature established a high level of patient satisfaction in the reported use of internet-based sources in seeking this self-education.1 This satisfaction largely stems from the convenience of immediate information retrieval utilizing internet-based search queries. This convenience led to patients with a proactive approach to their own healthcare and ultimately greater involvement in making patient centered medical decisions with their healthcare providers.2

The concept of healthcare literacy is tied closely with the health of an individual. Poor healthcare literacy was associated with poor self-reported health conditions, increased risk for hospital admissions, and greater healthcare costs.3–5 However, the ever-evolving material found through the internet can be difficult to condense in a manner applicable for the worldwide audience to work efficiently into their own healthcare literacy.6 The concept of the readability of patient education material serves an important role in a person’s ability to comprehend the material and plays a direct role in their healthcare literacy.7 It is recommended that the readability of patient education materials should not be higher than sixth-to-eighth-grade reading level.8 Moreover, the National Institutes of Health recommended the readability of self-administered patient questionnaires to be written no more than a sixth-grade reading level.9 However, the implementation of this readability recommendation is more difficult with regard to the internet due to lack of peer-review and regulatory factors which can aid in the authentication and validity of online patient educational materials.10

This imbalanced relationship between the immense usage of the internet and readability recommendations highlighted the necessity for further understanding of this climate of online information.11 The current literature, which investigated the readability of online patient education materials, have shown failure to meet these established readability recommendations, and it was supported that current healthcare literacy must be improved upon to improve healthcare outcomes holistically. 12–14

Within the field of endocrinology, the use of information in real-time is key for providers to guide the management of patients with endocrine and metabolic diseases and has led to greater advancement of patient care technology (i.e., eHealth apps, advancements in continuous glucose monitors).15–19 Despite this, there was a paucity of data which provided a comprehensive readability assessment of endocrine-related care. Moreover, current endocrine readability literature often was isolated to pathology or metabolic conditions.20–28 This limited real time comparative analytic data which can help endocrine providers in providing improved health care education.29–30 Therefore, the aim of this study was to assess contemporary online patient education material in endocrine disorders and management of care.

METHODS

Ethics

The data utilized in this study was entirely available for public use and did not involve human subjects. Therefore, Institutional Review Board approval was not required for this study.

Screening

Between November 2021 and January 2022, online education materials were extracted from Google® search queries (Google Inc., Mountain View, CA) for endocrine-related content areas of interest. Among these queries, three primary categories of content were of interest: adrenal, diabetes, and thyroid. These three categories of endocrine-related content were chosen due to perceived paucity of literature to date. Within each category, there were 10 common search items relevant for the category which were used in the queries as outlined in Table 1. These search items were chosen based on review of the most recent literature produced by the World Health Organization for the three categories.31–33

Table 1

Endocrine-related content areas of interest.

Adrenal	Diabetes	Thyroid
Addison’s Disease	Type 1 Diabetes	Graves’ Disease
Cushing’s Disease	Type 2 Diabetes	Hashimoto’s Thyroiditis
Cushing’s Syndrome	Insulin	Thyroid Hormone
Conn’s Syndrome & Hyperaldosteronism	Metformin	Thyroid Cancer
Congenital Adrenal Hyperplasia	Dipeptidyl Peptidase-4 (DPP-4) Inhibitor	Goiter
Pheochromocytoma	Sodium/Glucose Cotransporter (SGLT-2) Inhibitor	Levothyroxine
Multiple Endocrine Neoplasia	Insulin Pumps	Hypothyroidism
Neuroblastoma	Sulfonylureas	Hyperthyroidism
Paraganglioneuroma	Diabetic Ketoacidosis	Thyroid Biopsy
Hirsutism	Maturity Onset Diabetes of the Young	Thyroid and Iodine

Each search item was entered individually as a search query and the first 50 site link results which met the study’s inclusion and exclusion criteria were utilized in the analysis. For the purpose of this study, sites were included if they contained content that was pertinent towards providing general information on the search query of interests, as evaluated by the screeners (SPS, SS, KQ, SM). Sites met inclusion if they were in the English language, contained over 250 words minimum, and publicly available without any form of subscription required. Sites were excluded if they underwent a formal peer-reviewed research process with scientific indexing, the site explicitly specified the intended audience is for healthcare providers only, nonfunctioning search links, duplicate links, and/or did not meet the inclusion criteria.

Readability Quantification

Upon screening, the site content was reformatted to plain text in Microsoft Word®, as shown in previous literature methodology, to create efficient readability calculations later in the study.34–38 During the reformat phase, content material was removed for plain text if the screeners identified the content was unrelated to patient education. This specifically included removal of acknowledgments, author information, copyright disclaimers, figures and related captions and legends, references, and any web page navigation text. Moreover, the remaining content was unchanged from its site’s original format when converted to individual plain text documents for each site link.

After the reformat phase, each plain text document was evaluated quantitatively for its readability. This was performed through seven readability quantification measurements: Flesch Reading Ease (FRE), Flesch-Kincaid Grade Level (FKGL), Gunning Fog Index Readability Formula (FOG), Simple Measure of Gobbledygook Index (SMOG), Coleman-Liau Index (CLI), automated readability index (ARI), and Linsear Write Formula (LWF). The FRE was utilized in this study as it was one of the oldest and most used readability quantification measurement scales. FRE assesses the readability of the plain text using a scale of 0 to 100 where the higher the scaled number implies a higher readability of the plain text. For categorization purposes, the FRE in this study was scaled based off previous literature: very difficult (0–29), difficult (30–49), fairly difficult (50–59), standard (60–69), fairly easy (70–79), easy (80–89), and very easy (90–100).29,34–38

Similarly, the FKGL attempts to quantify the plain text by focusing on the average number of words per sentence and average number of syllables per word in the scale to correlate to a grade level (i.e., score of 9.4 would suggest a U.S. ninth grade reading level).39 The SMOG scale focuses on the total polysyllabic word count of the plain text to correlate to a grade level (i.e., a SMOG of 40 would approximate a U.S. ninth grade reading level). The CLI scale focuses on the average number of characters and sentences per 100 words of plain text (i.e., a CLI of 9.5 would correlate to a U.S. ninth to tenth grade reading level). The ARI is the summation of word and sentence difficulty to quantify a reading level utilizing similar characters per word and words per sentence (i.e., an ARI of 9 would approximate a U.S. ninth grade reading level). The LFW scale focuses on per 100 word sets similar to CLI, but also categorizes syllable counties per word as “easy words” (two or less syllables) or “hard words” (three or more syllables).27–29,39–40

The date of the search queries was recorded to limit potential ambiguity in search comparisons. Additionally, the country of origin of the site link was recorded (i.e., United States, United Kingdom). All data were recorded using Microsoft Excel® (Microsoft Corporation, Redmond, WA).

Statistical Analysis

Statistical analysis of the seven readability quantification scales was performed using Stata 14 Statistical Package® (StataCorp, College Station, TX) for descriptive statistics on the variables of interest, including counts, percentages, means, and standard deviations, where appropriate. Confidence intervals (CI) for parametric distribution were set at 95%. One way analysis of variance (ANOVA) was performed to compare average FRE measurements among search items with each category. The level of significance was set at p < 0.05.

RESULTS

Between November 2021 and January 2022, a total of 1,500 education materials (500 diabetic, 500 adrenal, and 500 thyroid education materials) were quantified for all seven readability assessment measurements for a total of 10,500 calculated measurements. The origin of 88.2% of all education material was the U.S. (n = 1,323), followed by 6.3% of articles from the U.K. (n = 94). The average grade reading level of all education materials was 13.08 (n = 1,500). The average grade reading levels and FRE of each topic of educational material were as shown in Table 2.

Table 2

Average grade reading levels across categories.

Content of Education Materials	Average Grade Reading Level	Confidence Intervals
Diabetes-Related	13.54	(CI: 13.08 – 14.00)
Thyroid-Related	12.91	(CI: 12.56 – 13.26)
Adrenal-Related	12.78	(CI: 12.34 – 13.22)
Content of Education Materials	Average Flesch Reading Ease Measurements	Confidence Intervals
Diabetes-Related	40.29 (“difficult to read”)	(CI: 34.45 – 46.13)
Thyroid-Related	39.82 (“difficult to read”)	(CI: 36.43 – 43.21)
Adrenal-Related	32.57 (“difficult to read”)	(CI: 28.17 – 36.97)

These results quantified all educational materials amongst the content categories as being “difficult to read” as per FRE measurements. Among the subgroup analysis of adrenal-related educational materials, no online education materials met at least one measurement of a sixth grade reading level or less (n = 50). In addition, content related to paraganglioneuromas had the highest grade reading level at 16.45 (CI: 15.01 – 17.89). Content related to neuroblastomas had the lowest grade reading level at 12.18 (CI: 11.84 – 12.54). The average grade reading level of all other adrenal-related content was as shown in Table 3.

Table 3

Reading level analysis of adrenal-related educational material.

Content Related To:	Grade Reading Level	Confidence Interval
Paraganglioneuroma	16.45	(CI: 15.01 – 17.89)
Conn’s Syndrome	14.74	(CI: 14.30 – 15.18)
Congenital Adrenal Hyperplasia	14.48	(CI: 14.05 – 14.91)
Pheochromocytoma	14.22	(CI: 13.81 – 14.63)
Multiple Endocrine Neoplasia	13.94	(CI: 13.58 – 14.30)
Cushing’s Disease	13.18	(CI: 13.14 – 13.22)
Addison’s Disease	13.16	(CI: 12.84 – 13.48)
Cushing’s Syndrome	13.12	(CI: 12.77 – 13.46)
Neuroblastoma	12.18	(CI: 11.84 – 12.54)

Among the analysis of FRE measurements for adrenal-related educational material, Addison’s Disease had the highest FRE measurement at 45.07 (CI: 30.76 – 39.38), which would qualify as “difficult to read”. Likewise, Conn’s Syndrome had the lowest FRE measurement at 21.97 (CI: 16.55 – 27.40), which would qualify as “very difficult to read”. The average FRE of all other adrenal-related content was as shown in Table 4.

Table 4

Average Flesch Reading Ease measurements of adrenalrelated education materials.

Content Related To:	Flesch Reading Ease Measurement	Confidence Intervals
Addison’s Disease	45.07 (“difficult”)	(CI: 30.76 – 39.38)
Neuroblastoma	42.81 (“difficult”)	(CI: 38.54 – 47.08)
Hirsutism	38.94 (“difficult”)	(CI: 33.75–44.13)
Cushing’s Disease	37.11 (“difficult”)	(CI: 32.96 – 41.25)
Multiple Endocrine Neoplasia	33.73 (“difficult”)	(CI: 29.87 – 37.59)
Congenital Adrenal Hyperplasia	29.32 (“very difficult”)	(CI: 23.77 – 34.87)
Pheochromocytoma	26.07 (“very difficult”)	(CI: 20.58–31.55)
Paraganglioneuromas	23.11 (“very difficult”)	(CI: 17.76 – 28.45)
Conn’s Syndrome	21.97 (“very difficult”)	(CI: 16.55 – 27.40)

Among the subgroup analysis of diabetes-related educational materials, 10% of Type 1 diabetes online education materials met at least one measurement of a sixth grade reading level or less (n = 5), and all others were less than 10%. In addition, content related to sodium/glucose cotransporter 2 (SGLT2) inhibitors had the highest-grade reading level at 14.70 (CI: 14.33 – 15.08). Content related to type 1 diabetes had the lowest grade reading level at 11.15 (CI: 11.85 – 11.45). The average grade reading level of all other diabetes-related content was as shown in Table 5.

Table 5

Average grade reading level of diabetes-related educational material.

Content Related To:	Grade Reading Level	Confidence Interval
Sodium/Glucose Cotransporter 2 (SGLT2) Inhibitors	14.70	(CI: 14.33 – 15.08)
Sulfonylureas	13.87	(CI: 12.99 – 14.76)
Maturity Onset Diabetes of the Young	13.54	(CI: 13.15 – 13.93)
Metformin	13.33	(CI: 12.96 – 13.71)
DPP-4 Inhibitors	13.29	(CI: 12.88 – 13.70)
Type 2 Diabetes	12.04	(CI: 11.10 – 12.99)
Insulin	12.03	(CI: 11.67 – 12.39)
Insulin Pumps	11.91	(CI: 11.56 – 12.26)
Diabetic Ketoacidosis	11.61	(CI: 11.26 – 11.96)
Type 1 Diabetes	11.15	(CI: 11.85 – 11.45)

Regarding FRE measurements for diabetes-related educational material, Type 1 diabetes had the highest FRE measurement at 52.28 (CI: 48.52 – 56.03), which would qualify as “fairly difficult to read”. Likewise, SGLT2 inhibitors had the lowest FRE measurement at 26.10 (CI: 22.80 – 29.39), which would qualify as “very difficult to read”. The average FRE of all other diabetes-related content was as shown in Table 6.

Table 6

Average Flesch Reading Ease measurements of diabetesrelated education materials.

Content Related To:	Flesch Reading Ease Measurement	Confidence Interval
Type 1 Diabetes	52.28 (“fairly difficult”)	(CI: 48.52 – 56.03)
Type 2 Diabetes	49.46 (“difficult”)	(CI: 46.35 – 52.58)
Insulin pumps	48.30 (“difficult”)	(CI: 43.94 – 52.67)
Insulin	47.32 (“difficult”)	(CI: 44.07 – 50.58)
Diabetic Ketoacidosis	44.91 (“difficult”)	(CI: 40.44 – 49.41)
Metformin	38.71 (“difficult”)	(CI: 34.36 – 43.06)
Maturity Onset Diabetes of the Young	34.69 (“difficult”)	(CI: 30.51 – 38.87)
DPP-4 Inhibitors	32.92 (“very difficult”)	(CI: 28.98 – 36.86)
Sulfonylureas	28.17 (“very difficult”)	(CI: 24.25 – 37.59)
SGLT2 Inhibitors	26.10 (“very difficult”)	(CI: 22.80 – 29.39)

Among the subgroup analysis of thyroid-related educational materials, 10% of hyperthyroidism online education materials met at least one measurement of a sixth grade reading level or less (n = 5), and all others were less than 10%. In addition, content related to hyperthyroidism had the highest-grade reading level at 13.73 (CI: 13.29 – 14.17). Content related to thyroid biopsy had the lowest grade reading level at 11.20 (CI: 10.86 – 11.54). The average grade reading level of all other thyroidrelated content was as shown in Table 7.

Table 7

Average grade reading level of thyroid-related educational material.

Content Related To:	Grade Reading Level	Confidence Interval
Hyperthyroidism	13.73	(CI: 13.29 – 14.17)
Graves’ Disease	13.18	(CI: 12.84 – 13.52)
Levothyroxine	12.97	(CI: 12.52 – 13.43)
Hashimoto’s Thyroiditis	12.92	(CI: 12.58 – 13.26)
Thyroid and Iodine	12.91	(CI: 12.51 – 13.30)
Thyroid Hormone	12.48	(CI: 12.14 – 12.82)
Hypothyroidism	12.44	(CI: 12.13 – 12.75)
Goiter	11.61	(CI: 11.32 – 11.89)
Thyroid Cancer	11.36	(CI: 11.04 – 11.68)
Thyroid Biopsy	11.20	(CI: 10.86 – 11.54)

Thyroid cancer had the highest FRE measurement at 47.95 (CI: 44.20 – 51.70), which would qualify as “difficult to read”. Likewise, hyperthyroidism had the lowest FRE measurement at 33.56 (CI: 29.15 – 37.97), which also would qualify as “difficult to read”. The average FRE of all other thyroid-related content was shown in Table 8.

Table 8

Average Flesch Reading Ease of thyroid-related content.

Content Related To:	Flesch Reading Ease Measurement	Confidence Interval
Thyroid Cancer	47.95 (“difficult to read”)	(CI: 44.20 – 51.70)
Thyroid Biopsy	47.80 (“difficult”)	(CI: 43.43 – 52.16)
Goiter	45.98 (“difficult”)	(CI: 42.50 – 49.46)
Thyroid Hormone	39.11 (“difficult”)	(CI: 35.23 – 42.99)
Thyroid and Iodine	38.35 (“difficult”)	(CI: 33.82 – 42.88))
Grave’s Disease	37.81 (“difficult”)	(CI: 33.83 – 41.79)
Hypothyroidism	37.51 (“difficult”)	(CI: 33.64 – 41.39)
Levothyroxine	36.72 (“difficult”)	(CI: 32.24 – 41.21)
Hashimoto’s Thyroiditis	33.56 (“difficult”)	(CI: 29.15 – 37.97)
Hyperthyroidism	33.44 (“difficult”)	(CI: 29.15 – 37.72)

DISCUSSION

The internet remains a form of a “pseudo-provider” due to its essential usage in nearly a third of information-seeking individuals when trying to self-diagnose or manage care without professional consultation.41–43 The growth of information-seeking behavior among individuals created both a beneficial effect in allowing extremely efficient dissemination of information than seen in previous years.44 This allowed a greater number of individuals to become empowered and gain greater awareness, including in healthcare literacy. In addition, that information also may aid in alleviating patient anxiety and involved with risk reduction strategies.45–49 However, this rise in information-seeking behavior has been suggested to cause cognitive changes in our ability to comprehend material and memory, and improper information seeking behavior may be related to risk behaviors including improper drug usage and potential addictions.45,47–50 The rise in these neuropsychological changes make it critical that individuals must be exposed to appropriate, legitimate comprehension to protect themselves.

The findings were in concordance with prior studies on diabetes and management. Moreover, a readability analysis on monogenic diabetes noted search items included in their analysis had failed to meet recommendations of a grade reading level standards of less than the sixth grade.21 While the presented study recorded 10% of online education materials (n = 5) meeting this readability recommendation, this growth may be negligible given the three years or more since the Guan et al.21 publication. In addition, the average FRE measurement of Grave’s disease is in concordance with previous literature in 2014 by Edmunds et al.26 which employed a similar screening methodology. In this study, the FRE score of Grave’s disease educational articles was found to qualify as “difficult to read” by FRE measurement. In fact, the current Grave’s diseases FRE of 37.81 was lower than what was found in the Edmund’s et al.26 study. This was likely because the current study used a larger sample (n = 50) of online education materials which were dedicated to Grave’s disease materials in comparison (n = 20). Regardless, this finding further compounded the need to attempt to simplify the readability of Grave’s disease literature. To the best of our knowledge, there was no dedicated literature on the readability of adrenal-related endocrine care, so the findings of this study were novel without a comparison. Moreover, the lack of adrenal-related online patient education materials which met the grade level readability recommendations raised priority in emphasized improvement in these materials.8–9

This study had multiple aspects which strengthens its findings. For example, the screening methodology accounted for 1,500 total online education materials and 10,500 readability quantification measurements creating the largest sample related endocrine care to date. The employment of seven scales minimized any potential measurement bias between scales. Secondly, this sample size had characteristically included material which were not formally scientifically indexed to limit variation in grade level readability as the audience of literature in PubMed or other scientific indexes may not be intended for the general audience.

However, this study was not without its limitations. The methodology did not account for utilization of other internet search query programs other than Google®. Google® comprised over 90% of the internet search query market share in the past year,51–53 so the methodology was believed to cover a valid portion of relevant online education materials. The methodology also implemented a plain text reformat of all included online education materials. This suggested that the study failed to account for an illustration or digital materials which has been shown to aid in improving healthcare literacy as well as patient-physician discussions.53

In addition, the inclusion criteria focused only on online education materials which were written in English and not formally peer reviewed. Thus, the results of this study may not apply to online education material which was not written in English. However, the increased use of online language translation applications raised the potential to investigate if there are any discrepancies in readability amongst foreign language texts.54–56

Another potential concern was consideration with temporal changes in search engine trends and the presence of a potential “bubble effect”.57 Original literature on this form of selection bias was in the context of using internet search queries when screening literature for systematic reviews.58 This was to account for how search query programs may tailor search results specific to the user’s preferences. However, the ultimate purpose of a readability analysis is to focus on the online results of the general crowd, so various measures can be used to limit potential selection bias such as using a clear internet search cache or specifically establishing a period for which internet search queries are performed as seen in this study. Future directions to develop after this study may be to encompass a larger set of search terms (i.e., more than 30) and use search trend technology to collect the most relevant search queries by the general worldwide audience. In addition, a larger scope of endocrine care which includes pituitary-related online education material can validate current literature in the long term.30

CONCLUSIONS

Healthcare literacy remains an important driving factor in the prognosis of a patient’s health condition and overall quality of life. Online education materials will continue to be a convenient source of information which can be used in the endocrinologist-patient relationship. However, the lifetime longitudinal care of patients with endocrine and metabolic diseases requires a greater awareness that the current climate of online educational materials do not meet readability recommendations for appropriate comprehension of the general audience.