Poor Readability of AOSSM Patient Education Resources and Opportunities for Improvement.

BACKGROUND: Appropriate education on the disease processes associated with orthopaedic pathology can affect patient expectations and functional outcome. HYPOTHESIS: Patient education resources from the American Orthopaedic Society for Sports Medicine (AOSSM) are too complex for comprehension by the average orthopaedic patient. STUDY
DESIGN: Cross-sectional study.
METHODS: Patient education resources provided by the AOSSM were analyzed with software that provided 10 readability scores as well as opportunities for improving readability. The readability scores were compared with the recommended eighth-grade reading level.
RESULTS: A total of 39 patient education resources were identified and evaluated. The mean ± SD reading grade-level scores were as follows: Coleman-Liau Index, 12.5 ± 1.11; New Dale-Chall Readability Formula, 10.9 ± 1.37; Flesch-Kincaid Grade Level, 9.9 ± 1.06; FORCAST Readability Formula, 11.4 ± 0.51; Fry Readability Formula, 12.8 ± 2.79; Gunning Fog Index, 11.9 ± 1.37; Raygor Readability Index, 13.1 ± 2.37; Simple Measure of Gobbledygook, 12.3 ± 0.90; Automated Readability Index, 11.2 ± 1.18; and New Automated Readability Index, 10.6 ± 1.27. After averaging the reading grade-level scores, only 1 patient education resource was found to be written at an 8th- to 9th-grade level, and 14 (36%) were written above a 12th-grade level. All scores were significantly different from the eighth-grade level (P < .0065). The percentage of complex words and long words were 19.6% ± 2.67% and 41.4% ± 3.18%, respectively.
CONCLUSION: Patient education resources provided by the AOSSM are at a significantly higher reading level than recommended. Simple changes can drastically improve these scores to increase health literacy and possibly outcome.

The search for reliable information on orthopaedic injuries often leads to subspecialty-specific websites. With the proliferation of internet access, gathering disease-specific information is more convenient than ever.[4] Health literacy has been linked to patient expectation and satisfaction and continues to shape outcomes following treatments for orthopaedic conditions.[1,15,19] Many studies have demonstrated that the readability of patient education resources in various medical fields is too complex for the average patient. In fact, the National Institutes of Health recommends a reading level of sixth to eighth grade.[2,12,13] Studies evaluating the reading level of orthopaedic patient education resources have consistently shown the readability score to be above this recommended level.[5-7,11,14]

Readability is commonly assessed with the Flesch-Kincaid Grade Level score.[11,20] This score is easy to interpret and is incorporated into commonly used word processing software. However, there are other readability scales available, including the Coleman-Liau Index, New Dale-Chall Readability Formula, FORCAST Readability Formula, Fry Readability Formula, Gunning Fog Index, Raygor Readability Index, Simple Measure of Gobbledygook (SMOG), Automated Readability Index (ARI), and New Automated Readability Index (New ARI). These readability scales emphasize different metrics compared with the Flesch-Kincaid and are supported by the Centers for Disease Control and Prevention and the National Cancer Institute.[2,12]

The purpose of this study was to evaluate, with these scales, the readability of the American Orthopaedic Society for Sports Medicine (AOSSM) patient education resources. Additionally, we examined the scales for opportunities and examples on how to improve readability. Our hypothesis was that AOSSM patient education resources are more complex than the recommended reading level.

Methods

We conducted a search of the injury-preventing resources at STOP Sports Injuries (“Prevent Injuries,” http://www.STOPSportsInjuries.org). Patient education material was downloaded in June 2017. All patient education resources were written in English. The text of the patient education resource was reformatted into Microsoft Word document files; during the reformatting process, we removed all figures, disclaimers, acknowledgments, citations, references, and hyperlinks. The reformatted patient education resources were then analyzed for readability with Readability Studio Professional Edition 2015 (Oleander Software Ltd). The software provided data, including scores from the Coleman-Liau, New Dale-Chall, FORCAST, Fry, Gunning Fog, Raygor, SMOG, ARI, New ARI, and Flesch-Kincaid. Table 1 provides a summary of the assessments. The software provided additional data, including percentage of complex words (defined as ≥3 syllables) and percentage of long words (defined as ≥6 characters). These terms are defined by the software.

TABLE 1

Readability Assessments With the Formulas Used to Calculate Them

Readability Assessment	Formula	Description
Flesch-Kincaid	(0.39 × B) + (11.8 × W) – 15.59	B = mean number of syllables per wordW = mean number of words per sentence
SMOG	1.043 × P×30S + 3.1291	P = number of words with ≥3 syllablesS = number of sentences
Coleman-Liau	(0.0588 × L) – (0.296 × T) – 15.8	L = mean number of letters/wordT = mean number of sentences/100 words
Gunning Fog	0.4 × (W/S + 100 × P/W)	S = mean number of sentencesW = mean number of words/sentenceP = mean number of words with ≥3 syllables
New Dale-Chall	0.0496 × W/S + 0.1579 × U/W + 3.6365	W = mean number of wordsS = mean number of sentencesU = unfamiliar words
FORCAST	20 – SS/10	SS = number of single-syllable words in 150-word sample
Raygor	No formula—uses the mean number of sentences and syllables per 100 words to determine graph point corresponding to grade level
Fry	No formula—uses the mean number of sentences and long words per 100 words to determine graph point corresponding to grade level

SMOG, Simple Measure of Gobbledygook.

Descriptive statistics were generated using Microsoft Excel. All reading level scores were averaged to produce a reading grade level for each patient education resource. A 1-sample, 2-tailed t test was used to compare the readability scores with the eighth-grade reading level. Significance was set at P = .0065, based on Bonferroni correction with alpha = 0.05.

Results

A total of 39 patient education resources were identified and evaluated by the Readability Studio software (see Appendix Table A1). Readability scores are summarized in Table 2. Averaging the reading grade-level scores demonstrated that only 1 patient education resource was written at an 8th- to 9th-grade level (“Preventing Baseball Injuries”) and 14 (36%) were written above a 12th-grade reading level (Figure 1). Of 39 patient education resources, 28 (72%) could not be evaluated via Fry on the basis of too many complex words. Similarly, 30 (77%) could not be evaluated for Raygor because of the excessive number of long words. Comparison was made between the reading grade levels of the patient education resources and an eighth-grade reading level. All scores were significantly different from the eighth-grade level (P < .0065) (Table 2).

TABLE A1

Patient Education Resources Identified and Evaluated by the Readability Studio Software

Document	Coleman-Liau	New Dale-Chall		Flesch-Kincaid	FORCAST	Fry	Gunning Fog	Raygor	SMOG	ARI	New ARI	Mean Reading Grade	Complex Words,b %	Long Words,c %
		Range	Mean
ACL	13.1	13-15	14.0	11.1	11.7	14	13.4	17	13.4	13.3	12.3	13.5	19.30	43
Baseball	8.9	7-8	7.5	7.4	10.8	*	8.5	*	10	9.7	8.7	8.9	15.20	36.20
Basketball	11.2	11-12	11.5	9.1	11.2	*	11.5	*	11.8	10	9.1	11.1	19.20	39.60
Cheerleading	11.3	9-10	9.5	9.5	10.9	11	11.1	12	12.3	10.7	9.8	11.0	17.60	37.70
Coaches Tips for Parents	10.8	9-10	9.5	9.4	10.4	10	11.4	11	12.1	10.2	8.9	10.6	14.80	34.50
Concussion	13.3	11-12	11.5	10.3	12.1	*	12	*	12.3	13.2	12.5	11.9	21.50	44.50
Conditioning	14.3	11-12	11.5	11.4	12	16	13.7	*	13.8	12.9	12.3	13.2	22.40	45.50
Cycling	11.4	9-10	9.5	8.9	10.6	10	10.4	11	11.6	9.9	8.8	10.4	15.10	37
Dance	12.1	9-10	9.5	9.1	11.4	*	11.9	*	11.8	11.6	10.6	11.0	19.40	41.90
Encourage Success	11.8	9-10	9.5	8.8	10.4	10	11.6	11	12	10.1	9.3	10.6	17	36.20
Female Triad	13.5	11-12	11.5	11.8	11.9	17	14.3	17	13.8	13	12	13.9	22.40	43.10
Field Hockey	13.2	11-12	11.5	10.9	11.7	*	13.6	*	13.1	11.7	11.3	12.3	23.40	43
Figure Skating	13.5	13-15	14.0	9.6	11.9	*	11.4	*	11.8	11.7	11.6	12.0	21.20	46
Football	12.6	11-12	11.5	10.2	11.5	*	12.2	*	12.5	11.5	10.7	11.8	19.90	43.80
Golf	10.9	9-10	9.5	8.7	10.3	*	9.9	*	11.3	9.9	8.9	10.1	14.80	34.70
Gymnastics	13.2	11-12	11.5	10.1	11.7	*	12.2	*	12.2	11.6	11.2	11.8	22.10	42.50
Heat Illness	11.9	11-12	11.5	10.3	11.7	*	12.1	*	12.4	10.4	9.5	11.7	18.90	39.80
Hockey	12.3	11-12	11.5	9.9	11.5	*	12.8	*	12.6	11.2	10.4	11.8	20.20	42.90
Inline Skating	13.4	11-12	11.5	10.7	11.7	*	10.4	*	12.3	10.8	10.8	11.7	19.30	45.10
Instrumentalists	14.1	11-12	11.5	11.2	12.2	*	13.5	*	13.5	12.3	12	12.7	23.60	45.30
Knee Injuries	10.8	11-12	11.5	10	10.5	11	12.9	13	13	10.4	9	11.6	18.90	38.20
Lacrosse	13.1	11-12	11.5	10.3	11.1	13	11.9	13	12.9	12	11.3	12.1	19.60	40.10
Martial Arts	12.5	11-12	11.5	9.8	11.2	*	10.7	*	11.8	10.2	10.2	11.3	18.70	40.90
Overuse	12.8	11-12	11.5	10.8	12	*	12.8	*	13	10.9	10.6	12.2	22.00	44.40
Prevent Overuse Injuries	12.7	9-10	9.5	9.9	11.5	*	11.8	*	12.2	10.5	9.7	11.3	17.90	39.70
Rowing	12.4	11-12	11.5	9.9	10.9	12	11.2	13	11.9	11.3	10.5	11.6	15.70	38.80
Rugby	13.2	11-12	11.5	10.5	11.7	*	13.5	*	12.8	11.3	10.6	12.2	22.30	42.80
Running	11.9	7-8	7.5	8.1	11.3	*	9.6	*	10.5	10.6	10.3	9.8	16.80	41.30
Skiing and Snowboarding	13.6	11-12	11.5	9.6	11.9	*	12.5	*	11.7	11.5	11.6	11.8	24.20	43.60
Soccer	12.1	11-12	11.5	9.9	11.2	*	12	*	12.6	10.7	10.1	11.6	20.20	40
Softball	11.3	9-10	9.5	8.4	11.2	*	10.1	*	11	9.6	9.5	10.3	18.10	38.80
Sports Nutrition	13.3	11-12	11.5	12.1	11.8	*	14.1	*	14	12.9	12.2	12.8	23.50	44.60
Strength Training	15.2	11-12	11.5	11.7	11.5	17	14.4	*	13.8	14.3	13.8	13.6	22.80	45.70
Swimming	13.9	11-12	11.5	10.1	11.9	*	12.2	*	12.5	12.4	11.8	12.0	20	45.60
Tennis	11	9-10	9.5	8.2	11	*	10.2	*	10.9	9.9	9.1	10.1	14.90	41.20
Volleyball	13.3	11-12	11.5	10.4	11.8	8	12.8	*	12.4	11.4	11.1	11.5	22.20	43.70
Water Polo	13.2	11-12	11.5	10.2	11.7	*	11.3	*	12.3	10.9	10.8	11.7	21	41.80
When Play Is Too Much	11.7	9-10	9.5	8.8	11	*	11.3	*	11.8	9.6	8.9	10.7	18.70	37.90
Wrestling	13.4	11-12	11.5	10.5	11.5	*	12.9	*	13	12.5	11.9	12.1	21.10	43.50

Asterisks indicate that readability score was unable to be calculated due to too many long or high syllable words. ACL, anterior cruciate ligament; ARI, Automated Readability Index; SMOG, Simple Measure of Gobbledygook.

Complex words: ≥3 syllables (values in proportions).

Long words: ≥6 characters.

TABLE 2

Readability Data Produced by the Readability Studio Software

Readability Test	Mean ± SD	P Value
Flesch-Kincaid	9.9 ± 1.06	7.14E-14
SMOG	12.3 ± 0.90	4.71E-28
Coleman-Liau	12.5 ± 1.11	4.80E-24
Gunning Fog	11.9 ± 1.37	3.01E-20
New Dale-Chall	10.9 ± 1.37	7.09E-16
FORCAST	11.4 ± 0.51	3.17E-33
Raygor	13.1 ± 2.37	.000193522
Fry	12.8 ± 2.79	.000188962
ARI	11.2 ± 1.18	1.74E-19
New ARI	10.6 ± 1.27	2.42E-15

All scores compared with eighth-grade reading level by 2-tailed t test. Significance was set at P = .0065. ARI, Automated Readability Index; SMOG, Simple Measure of Gobbledygook.

Figure 1.

The reading grade level of all scales was averaged to provide a composite readability score. Only 1 patient education resource was written at an 8th- to 9th-grade level, and 14 (36%) were written above a 12th-grade reading level.

Total word count per patient education resource was 714 ± 143 words. The percentages of complex words and long words were 19.6% ± 2.67% and 41.4% ± 3.18%, respectively. Complex words were defined as words with ≥3 syllables, and long words were those with ≥6 characters. Each patient education resource contained a mean 6.90 overly long sentences. The longest sentence was 35.6 ± 6.18 words. Table 3 lists words that were deemed problematic by the software along with suggested alternatives to improve readability.

TABLE 3

Problem Words and Suggested Alternatives Produced by Readability Studio Pro Software

Problem Word	Suggested Alternative
Anterior	Front
Beneficial	Helpful
Debilitating	Weakening
Examination	Check
Rehabilitate	Restore
Physician	Doctor
Adjacent	Next to
Incorporating	Blending, joining, mixing
Opportunities	Chances
Compress	Squeeze
On a regular basis	Regularly
Strategies	Plans
Subsequently	After
Deplete	Empty
Beneficial	Helpful
Components	Parts
Abrasions	Scratches
Inadvertent	Careless
Utilization	Use
Prioritize	Rank
Lacerations	Cuts
Occurrence	Event
Typically	Often
Alteration	Change
Emphasize	Stress
Internal	Inner
External	Outer

Discussion

With the proliferation and nearly ubiquitous consumption of internet access, patients are looking for reliable online sources regarding medical care.[4] Internet searches can often lead to specialty sites such as the AOSSM. Multiple studies have shown that the readability score of patient education resources may be too complex for the average reader.[5-7,11,14] Our study reveals that the material found on the STOP Sports Injuries website is no different.

Previous studies have used the Flesch-Kincaid score to evaluate readability.[11] , 20 In the current study, we used multiple measures of readability in addition to Flesch-Kincaid, including Coleman-Liau, New Dale-Chall, FORCAST, Fry, Gunning Fog, Raygor, SMOG, ARI, and New ARI. All reported reading grade levels were significantly higher than the recommended eighth-grade reading level. Our results add to the validity of the Flesch-Kincaid scores previously reported.

Health literacy is critical in orthopaedic surgery. Studies have linked literacy to patients’ expectations and outcomes and have emphasized the importance of setting realistic expectations.[1,15,19] A significant portion of the patient education process lies in patient education resources, as studies have also shown that patients may not ask questions in the office or may express understanding when they are actually unsure.[8,10] Additionally, the increasing use of patient-reported outcome measures highlights the need for health literacy given the roles that patient expectation and satisfaction play.

Multiple studies have found that orthopaedic patient education resources are too complex, often well above the eighth-grade reading level.[5-7,11,14] Other fields of medicine have faced similar challenges and have attempted to simplify the language to improve readability.[3,17,18] Colaco et al[3] evaluated online urology patient education resources using 10 commonly used assessment tools and determined that most were written at an 11th-grade level or higher. Svider et al[18] concluded that online otolaryngology patient education resources were too difficult based on the recommended reading level. They used the same 10 assessment tools as in this study and also determined that most resources were at an 11th-grade level or higher. Both author groups concluded that patient education resources should be simplified to facilitate comprehension. Schoof and Wallace[17] reported on the readability of online family medicine patient education resources. Using an online program that focuses on sentence length and word frequency, the authors found that the majority of currently available family medicine patient education resources were at a sixth-grade level. Interestingly, they found that the number of resources at a sixth-grade level in 2012 had increased significantly since 2004 (59% vs 5%). These authors concluded that efforts should continue to be made to simplify and reduce the reading demand of patient education resources to support patient understanding and communication.

Our data indicate that word complexity and word length are driving the increased reading grade level for online patient education resources for orthopaedic sports medicine. Orthopaedic surgery involves the use of complex terms to describe anatomic structures as well as mechanical principles. This jargon can be intimidating, especially for those untrained in medicine. Orthopaedic surgeons undergo years of specialized training to fully understand the anatomy and disease process; attempting to condense this into generalizable and understandable concepts is a challenge. Even in the medical field, musculoskeletal education was shown to be lacking during medical school.[9,16] Nonetheless, we provide an example of simplifying language with the use of more common terms to refer to anatomy, as well as simpler descriptions in shorter sentences. Figure 2 provides a comparison between the original text and the modified version. The modified version produces a reading grade level lower in all indices except Gunning Fog (Table 4). Here we demonstrate that by making a few simple changes, the reading grade level can be brought to an appropriate level.

Figure 2.

An excerpt from a section on anterior cruciate ligament (ACL) injury prevention. The text was edited with the use of simpler language as recommended by Readability Studio Pro software to produce more accessible reading scores.

TABLE 4

Readability Scores Comparing the Original Text and the Revised Text

	Original	Edited
Flesch-Kincaid	11.1	6.2
SMOG	13	10.5
Coleman-Liau	13.1	7.9
Gunning Fog	13.4	15.5
New Dale-Chall	14	7.5
FORCAST	11.7	10.4
Raygorb	—	—
Fryb	—	—
ARI	13.3	5
New ARI	12.3	4.4
Mean	13.3	7.4

Nearly every readability assessment revealed an improved score. ARI, Automated Readability Index; SMOG, Simple Measure of Gobbledygook.

Test could not be completed because of the short text.

There are several limitations to this study. While there is no gold standard measurement for readability, we chose to use multiple scales that emphasize different aspects of readability to improve our validity. Additionally, the analysis evaluated the text of the patient education material and excluded material such as images, tables, and animations, which may prove helpful. Moreover, while metrics such as syllables per word, words per sentence, and word length may influence readability as measured by software, this does not necessarily correlate with comprehension. The strengths of this study include the use of sophisticated software to provide data on complex and long words. Given the use of multiple comparisons, a Bonferroni correction was used for significance.

The overall goal is for better patient understanding of the disease-specific condition or treatment, and readability of patient education resources is only one factor toward that end. Thus far, there is no consensus on the minimum readability of patient education resources to have a meaningful impact on patient understanding. This study shows that with modification, patient education resources can be provided at an appropriate reading grade level to promote understanding and to ultimately inform medical decisions and better define expectations. Other surgical specialties have noted reading levels of online patient education resources to be too high and are simplifying the language to improve understanding and communication.[3,18] Our study demonstrates that the readability of online patient education resources in orthopaedic sports medicine remains higher than recommended; however, efforts to simplify language can yield a lower reading demand. Providers bear the responsibility of ensuring adequate patient education as well as removing barriers toward that goal. Ultimate outcome can be influenced by patient expectations and satisfaction, and the readability of patient education resources is a modifiable factor in improving care.

Conclusion

Patient education resources provided by the AOSSM are written above recommended reading grade levels. Small changes can produce significant improvement in the readability scores to support increased health literacy. Ultimately, this may improve patient understanding and expectations and lead to improved satisfaction and outcomes.