Readability assessment of patient educational materials for shoulder arthroplasty from top academic orthopedic institutions.

BACKGROUND: Previous studies have demonstrated that online patient educational materials are written at reading levels too advanced for the average patient. The average American reads at the eighth-grade reading level. To date, the readability of online educational material of academic centers for shoulder arthroplasty has not been analyzed.
METHODS: Online patient educational materials from the top 25 orthopedic institutions, as ranked by U.S. News & World Report, were assessed utilizing the following readability assessments: Flesch-Kincaid (FK), Flesch Reading Ease, Gunning Fog Index, Coleman-Liau Index, Simple Measure of the Gobbledygook Index, Automated Readability Index, FORCAST, and the New Dale and Chall Readability. All of these scores, with the exception of the Flesch Reading Ease, provide an output indicating reading difficulty based on grade level. Correlations between academic institutional ranking and FK scores were evaluated using a Spearman regression. Lastly, additional factors including geographical location, private versus public institution, and use of concomitant multi-media modalities that may impact institutional readability scores (as determined by FK) were evaluated.
RESULTS: Only 16.0% of the top 25 institutions included online material at or below the eighth-grade reading level. Moreover, half of the online resources evaluated (those with FK score ≥9.3) were not at a suitable reading level for more than two-thirds of the general United States population (∼70%). Overall, the composite mean scores were 9.5 ± 2.1 for FK, 52.8 ± 9. for 4 Flesch Reading Ease, 12.2 ± 2.4 for Gunning Fog, 11.6 ± 1.8 for Coleman-Liau, 12.3 ± 1.7 for Simple Measure of the Gobbledygook Index, 9.6 ± 2.6 for Automated Readability, 11.1 ± 0.6 for FORCAST, and 5.9 ± 0.6 for New Dale and Chall. There was no correlation between institutional ranking and FK scores (ρ = -0.15; P = .946). Geographical location, private versus public institution, and use of concomitant multi-media modalities were not significantly associated with readability.
CONCLUSION: Shoulder arthroplasty online patient educational material at top-ranked orthopedic institutions have poor readability and are likely not suitable for the majority of patients in the United States.

Technological advancements have vastly enhanced dissemination of educational resources to patients. This is especially true during the current COVID era, where patients are more dependent on online educational medical resources than ever before.,,,, Although accessing this information may be easy, the true value of online resources is determined by the readers’ health literacy and ability to comprehend the content of the resource. This is supported by previous research demonstrating strong associations between low health literacy and increased hospitalization rates, as well as diminished overall outcomes in a multitude of fields.,,,, This relationship is of significant importance as the average reading ability in the United States approximates an eighth-grade reading level., Therefore, it is suggested that online educational material should be written at a sixth-grade reading level in an effort to increase health literacy, enhance informed consent, and promote better outcomes.

The number of studies investigating the readability of online health educational resources has increased, in an effort to improve the quality of patient care. By assessing readability, clinicians can critically evaluate the educational materials readily accessible to patients as well as provide an opportunity to advocate for education material that is more inclusive to all patients. Importantly, academic orthopedic centers are one of the primary contributors to accurate online information for patients. Expectedly, previous studies have analyzed the readability of educational resources from leading orthopedic organizations,,,,,, and orthopedic academic centers for various pathologies., However, to the authors’ knowledge, no previous study has investigated the readability of educational materials of academic centers for shoulder arthroplasty.

The purpose of this study was to investigate the readability of patient online educational materials specific to shoulder arthroplasty at the top-ranked orthopedic institutions in the United States. It was hypothesized that online educational material for shoulder arthroplasty is above the reading level of the majority of patients in the United States.

Methods

The U.S. News and World Report of orthopedic specialty rankings were used to identify the top 25 academic orthopedic institutions in the United States in 2021. Online patient educational materials for shoulder arthroplasty, including both total shoulder arthroplasty and reverse total shoulder arthroplasty, were searched and collected for each institution. Institutions were excluded from analysis if no online patient educational material for shoulder arthroplasty was present. The patient education resources were converted into text-only format to exclude figures, disclaimers, acknowledgements, citations, references, and hyperlinks. Reformatted patient education files were then analyzed using ReadablePro 2021 (Readable, Added Bytes Ltd.; Horsham, UK).

Statistical analysis

Utilizing this software, the following readability test scores were calculated: Flesch-Kincaid (FK), Flesch Reading Ease, Gunning Fog Index, Coleman-Liau Index, Simple Measure of the Gobbledygook Index (SMOG), Automated Readability Index, FORCAST, and the New Dale and Chall Readability. Specific equations for these readability metrics have been detailed in previous studies and are displayed in Table I., All of these scores, with the exception of the Flesch Reading Ease, provide an output indicating reading difficulty based on grade level (eg, score of 8 equates to 8th-grade reading level). A linear regression analysis was used to generate variance inflation factors, with values ≥ 10 indicating collinearity between various readability scores.

Table I

Readability assessments and associated formulas.

Readability assessment	Formula
Flesch-Kincaid	(0.39xmean#ofsyllablesperword)+(11.8xmean#ofwordspersentence)
Flesch Reading Ease	206.835−(1.015xmean#ofwordspersentence)−(84.6xmean#ofsyllablesperword)
Gunning Fog Index	0.4x(mean#ofwordsmean#ofsentences+100xmean#ofwordswith ≥3syllablesmean#ofwords)
Coleman-Liau Index	(0.0588xmean#oflettersword)−(0.296xmean#ofsentences100words)
Simple Measure of the Gobbledygook Index	1.043x(#ofwordswith ≥3syllables)x(30#ofsentences)+3.1291
Automated Readability Index	4.71(letterswords)+0.5(wordssentences)−21.43
FORCAST	20−(#ofsinglesyllablewordsin150wordsample10)
New Dale and Chall Index	0.0496x(mean#ofwordsmean#ofsentences)+0.1579x(unfamiliarwordsmean#ofwords)+3.6365

Continuous variables were presented as means and standard deviations. Correlation between academic institutional ranking and FK scores were evaluated using a Spearman regression. Additional factors including geographic location (urban versus rural), private versus public institution, and use of concomitant multi-media modalities (pictures or videos present on institutions website versus no media) that may impact institutional readability scores (as determined by FK) were analyzed with independent t-test and Mann-Whitney tests for parametric and nonparametric continuous variables, respectively. All tests were 2-sided. Analyses were performed with SPSS 27.0 (IBM Corp., Armonk, NY, USA).

Results

In total, 23 of top 25 institutions had educational patient resources that could be utilized for analysis. Readability scores were calculated for the top 23 institution’s online resources for total and reverse total shoulder arthroplasty. A wide spectrum of FK scores were observed, ranging from 6.6 to 15.4. Notably, only 17.3% of the institutions included in this analysis had online material at or below the eighth-grade reading level (Figure 1). Overall, the composite mean scores were 9.5 ± 2.1 for FK, 52.8 ± 9.4 for Flesch Reading Ease, 12.2 ± 2.4 for Gunning Fog, 11.6 ± 1.8 for Coleman-Liau, 12.3 ± 1.7 for SMOG, 9.6 ± 2.6 for Automated Readability, 11.1 ± 0.6 for FORCAST, and 5.9 ± 0.6 for New Dale and Chall.

Figure 1

Flesch-Kincaid grade level readability scores for online shoulder arthroplasty resources at top orthopaedic institutions relative to average United States reading level.

When assessing multicolinearity, it was determined that all scores (excluding Flesch-Kinaid Reading Ease score) demonstrated a high degree of collinearity relative to the FK score (variance inflation factor for each score: Gunning Fog Index = 447.1, Coleman-Liau Index = 36.0, SMOG = 59.8, Automated Readability Index = 18.8, FORCAST = 12.4, New Dale and Chall Index = 357.05).

Individual readability scoring metrics for each institution can be appreciated in Table II. Considering the high degree of collinearity between scores, subsequent analyses focused on FK scores. There was no correlation between institutional ranking and FK scores (ρ = −0.15; P = .946). The following variables did not demonstrate a significant relationship with institutional online resource readability based on FK scores: geographic location (urban: 9.5 versus rural: 8.8, P = .727), private versus public institution (9.4 versus 9.78, P = .758), and use of concomitant multi-media modalities (presence of pictures or videos: 9.4 versus no media: 9.7, P = .741).

Table II

Individual reading scores for online shoulder arthroplasty resources at top orthopedic institutions.

Hospital rank	Flesch-Kincaid	Flesch reading ease	Gunning Fog index	Coleman-Liau index	Simple Measure of Gobbledygook index	Automated readability index	FORCAST	New Dale and Chall readability
1	11.2	44.1	15.4	12.7	14.2	11.1	11.5	6.7
2	11.2	44.1	15.4	12.7	14.2	11.1	11.5	6.7
3	8.6	53.0	10.5	12.5	11.4	8.8	11.5	5.5
4	10.1	49.2	13.2	12.0	12.8	9.9	11.0	6.2
5	9.3	55.3	12.1	11.6	12.7	9.7	10.9	5.9
6	10.7	47.5	14.0	12.3	13.5	10.8	11.1	6.3
7	8.2	55.3	9.2	11.8	10.7	8.4	11.6	5.2
8	8.2	55.3	9.2	11.8	10.7	8.4	11.6	5.2
9	9.2	51.1	11.4	11.7	11.9	8.7	11.1	5.7
10	6.1	71.5	8.6	8.6	9.7	5.8	9.8	5.1
11	8.8	61.0	11.6	9.7	11.6	8.5	10.3	5.8
12	10.4	52.0	12.5	11.5	12.5	10.6	11.3	6.0
13	7.0	59.1	10.1	9.7	10.2	6.0	10.8	5.5
14	7.0	59.1	10.1	9.7	10.2	6.0	10.8	5.5
15	10.8	49.3	13.7	12.2	13.6	11.4	11.0	6.3
16	10.9	47.0	13.5	13.2	13.3	11.6	11.6	6.2
17	12.1	45.7	13.8	13.5	13.0	13.7	11.8	6.2
18	9.4	54.5	11.8	10.7	12.0	8.9	11.1	5.8
19	15.4	24.5	18.6	16.8	17.3	17.0	12.6	7.4
20	6.6	67.8	9.6	9.4	10.1	6.5	10.3	5.3
21	8.9	58.0	11.5	10.5	12.0	8.7	10.3	5.7
22	8.8	61.0	11.7	9.7	11.6	8.6	10.3	5.8
23	10.1	49.4	13.2	12.3	13.0	10.2	11.2	6.1
Mean (SD)	9.5 (2.1)	52.8 (9.4)	12.2 (2.4)	11.6 (1.8)	12.3 (1.7)	9.6 (2.6)	11.1 (0.6)	5.9 (0.6)

SD, standard deviation.

Beyond this, 17.4% of institutional websites had available educational photos and 36.4% had videos. Focusing on institutional websites with pictures, the average number of photos was 3.0 ± 2.4. Similarly, of the institutional websites with videos, the average number of videos was 1.25 ± 0.46. Upon analyzing our institutions online patient resources, the FK score was 11.3, placing our institution in the bottom quartile in terms of readability.

Discussion

The major finding of this study is that currently only 17.3% of the top orthopedic institutions provided online patient educational materials for shoulder arthroplasty at a suitable reading level (below 8th-grade reading level) to provide value to patients. Academic orthopedic institutional ranking did not have any correlation with the readability of online patient educational materials (ρ = −0.15; P = .946). The scores utilized in this study demonstrated a high degree of collinearity, indicating that irrespective of the score being interpreted, educational resources across most institutions were too difficult to read based on estimated grade levels (eg, mean FK of 9.5). Geographic location, private versus public, and use of concomitant multi-media modalities did not appear to have a significant association with readability. Lastly, only a fraction of the top institutions provided pictures or videos as supplemental educational resources on their online websites (17.4 and 36.4%, respectively).

These findings are in accordance with other domains of orthopedic surgery, which have also demonstrated poor readability for patient educational materials.,,,,,,,, In the setting of hip preservation–related educational materials, Parsa et al identified a statistically weak negative correlation between orthopedic institutional rank (ρ = −0.21) and readability. Conversely, the current study was not able to confirm these notions. Additionally, institution-related factors including geographic location and private versus public sector were not found to significantly impact readability scores. Instead, disseminated educational resources may be particular to an institutions’ individual initiative to provide resources with appropriate levels of readability for their patient population.

Surprisingly, only a minority of the institutions provided educational photos (17.4%) or videos (36.4%) for shoulder arthroplasty. Previous research utilizing a novel video assessment tool has demonstrated that when institutions do provide videos, they are usually of high quality in terms of content. This appears to be a major missed opportunity as additional supplemental resources including pictures and videos may be a key secondary mediator to increase health literacy in the setting of poor readability.

It appears at this current time, few institutions have fully incorporated all of these aspects to ensure the best health literacy for their patients. In fact, analysis of our own institution revealed an ample amount of pictures (9), but a FK score of 11.3. This would place our institution in the bottom quartile of readability relative to this current study cohort. Some key suggestions to improve readability are as follows: using shorter words with fewer syllables, using fewer words per sentence, and incorporation of more visual supplemental material.

Ultimately, websites will derive the greatest benefit by substituting exhaustive explanations for brief descriptions that improve reader comprehension. This ideology seems to be the key to improving readability scores for online patient educational material for total and reverse total shoulder arthroplasty. Anecdotally, institutions with poorer readability scores commonly presented specific and complex information regarding anatomic references, procedural steps, anesthesia types, and implant designs and materials. Conversely, institutions with the best scores simplified these discussions, referencing the humerus as the “upper arm bone,” the glenoid as the “socket of the shoulder,” and implants as a “metal ball attached to a stem and socket.” Additional common and difficult terms can be found in Table III with suggested alternatives. Overall, exchanging medical jargon for oversimplified descriptions and exclusion of technical aspects of the procedure will greatly enhance readability.

Table III

Identification of commonly used difficult terms related to total and reverse total shoulder replacement and suggested alternatives.

General	Alternatives	Anatomy	Alternatives
Anesthetic	Pain reducing	Articular	Joint surface
Arthropathy	Joint disease	Axillary nerve	Nerve of underarm region
Arthroplasty	Replacement	Deltoid	Shoulder muscle
Avascular	Lack of blood supply	Glenohumeral	Shoulder joint
Component	Part	Glenoid	Socket of shoulder
Debridement	Joint cleaning	Humerus	Upper arm bone
Degenerative	Loss over time/wear and tear	Scapula	Shoulder blade
Hemiarthoplasty	Partial replacement	Synovial fluid	Joint fluid
Instruments	Tools
Irreparable	Cannot be repaired	Implant	Alternatives
Meticulous	Careful	Artificial	Manmade
Necrosis	Tissue death	Cobalt	Metal
Osteointegration	Bone attachment to implant	Polyethylene	Plastic
Osteoarthritis	Arthritis	Prosthesis/prosthetic	Implant
Osteonecrosis	Dead bone	Titanium	Metal
Strenuous	Hard

Limitations

This study has several noteworthy limitations. First, readability formulas do not fully capture all factors that may also affect the patient’s ability to comprehend educational materials such as shorter unfamiliar words (medical jargon) and shorter sentences with increased complexity. Additional studies are required to determine the validity of the patient educational material, while also measuring the patient’s comprehension of the material to accurately determine how well patients understand material presented in these formats. Furthermore, readability assessments do not account for educational benefit provided by visual supplemental materials such as pictures and videos. Lastly, this study utilized rankings provided by US News and World Report of orthopedic specialty rankings, which may not be the most accurate ranking system. Moreover, the educational material for shoulder arthroplasty from these institutions may not be representative of those across all orthopedic institutions.

Conclusion

This analysis has demonstrated that online patient educational materials for shoulder arthroplasty provided by the nation’s top orthopedic institutions, as well as our own, have poor readability. It is important to acknowledge this opportunity for improvement and strive to produce patient education materials with lower grade level readability standards, preferably at the eighth-grade level or below. This will ensure a higher degree of health literacy and ultimately lead to improved patient outcomes. Finally, this analysis revealed only a minority of institutions included supplemental educational pictures and videos. Future online patient educational materials should strive to include a greater number of supplemental materials as a secondary method to improve health literacy.

Disclaimers

Funding: The Health Center/UConn Musculoskeletal Institute has received direct funding and material support from Inc. (Naples, FL, ). The company had no influence on study design, data collection, or interpretation of the results or the final manuscript.

Conflict of Interest: Dr Mazzocca reports research grants and is a consultant for , Inc. The other authors, their immediate families, and any research foundation with which they are affiliated have not received any financial payments or other benefits from any commercial entity related to the subject of this article.