Can Patients Read, Understand, and Act on Online Resources for Anterior Cruciate Ligament Surgery?

Background: Patients undergoing elective procedures often utilize online educational materials to familiarize themselves with the surgical procedure and expected postoperative recovery. While the Internet is easily accessible and ubiquitous today, the ability of patients to read, understand, and act on these materials is unknown. Purpose: To evaluate online resources about anterior cruciate ligament (ACL) surgery utilizing measures of readability, understandability, and actionability. Study Design: Cross-sectional study; Level of evidence, 4.
Methods: Using the term "ACL surgery," 2 independent searches were performed utilizing a public search engine (Google.com). Patient education materials were identified from the top 50 results. Audiovisual materials, news articles, materials intended for advertising or medical professionals, and materials unrelated to ACL surgery were excluded. Readability was quantified using the Flesch Reading Ease, Flesch-Kincaid Grade Level, Simple Measure of Gobbledygook, Coleman-Liau Index, Automated Readability Index, and Gunning Fog Index. The Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P) was utilized to assess the actionability and understandability of materials. For each online source, the relationship between its Google search rank (from first to last) and its readability, understandability, and actionability was calculated utilizing the Spearman rank correlation coefficient (ρS).
Results: Overall, we identified 68 unique websites, of which 39 met inclusion criteria. The mean Flesch-Kincaid Grade Level was 10.08 ± 2.34, with no website scoring at or below the 6th-grade level. Mean understandability and actionability scores were 59.18 ± 10.86 (range, 33.64-79.17) and 34.41 ± 22.31 (range, 0.00-81.67), respectively. Only 5 (12.82%) and 1 (2.56%) resource scored above the 70% adequate PEMAT-P threshold mark for understandability and actionability, respectively. Readability (lowest P value = .103), understandability (ρS = -0.13; P = .441), and actionability (ρS = 0.28; P = .096) scores were not associated with Google rank.
Conclusion: Patient education materials on ACL surgery scored poorly with respect to readability, understandability, and actionability. No online resource scored at the recommended reading level of the American Medical Association or National Institutes of Health. Only 5 resources scored above the proven threshold for understandability, and only 1 resource scored above it for actionability.

Ruptures of the anterior cruciate ligament (ACL) are common, with an estimated incidence of 43.5 ACL injuries occurring per 100,000 person-years. Reconstruction of the ACL has proven to effectively restore knee stability and allow patients to return to athletic activity.

Patients undergoing elective procedures often utilize online educational materials to familiarize themselves with the surgical procedure and expected postoperative recovery. While the Internet is easily accessible and ubiquitous in today’s society, previous studies have demonstrated that information regarding various orthopaedic procedures varies widely in quality and readability. Furthermore, the medical information found online has been shown to directly influence patient health care decision-making. To be helpful to patients, online educational materials should be written at an elementary school reading level in a format that is understandable with actionable direction that positively affects health care interactions. The American Medical Association (AMA) and the National Institutes of Health (NIH) recommend that patient education materials be written at a 6th-grade reading level.

Previous orthopaedic and surgical literature has analyzed the readability of patient education materials to assess their reading grade level. Outside of their mathematical description, readability measures have long had a nebulous definition. One of the early measures of readability was invented and refined by Rudolf Flesch in the 1940s. Flesch defined his measure of “readability” as “a statistical formula for the objective measurement” of “comprehension difficulty.” Since the advent of this readability measure, many similar readability formulas have been created that utilize mathematical formulas and variables such as syllables per word and number of words in a sentence to measure how difficult a text is to comprehend, with some measures attempting to define a reading grade level to assist evaluators gain a further sense of the reading difficulty level.

In addition to being nebulous in their definition, readability measures are limited in their assessment of a material’s ability to convey information such that readers can process and act on key messages. This limitation has been recognized, and the Patient Education Materials Assessment Tool (PEMAT) has become increasingly used to study patient education materials in surgical fields. The PEMAT is used to evaluate the “understandability” and “actionability” of educational materials. Understandability is defined as the ability of readers to “process and explain key messages,” and actionability is defined as the ability of readers to “identify what they can do based on the information presented.” However, there is a paucity of literature evaluating whether online patient education resources on ACL reconstruction are presented so that readers can efficiently understand or identify available actions.

The purpose of the current study was to utilize the PEMAT and validated readability algorithms to quantify the readability, understandability, and actionability of online educational materials related to ACL surgery. We hypothesized that when used to calculate the readability, understandability, and actionability of online educational resources, these resources would not meet the minimum AMA/NIH recommendations and PEMAT thresholds.

Methods

Identification of Educational Materials

Online patient education materials were identified using the Google.com search engine. On October 14, 2019, the average 5-year popularity of the following keywords was compared using Google Trends : “ACL surgery,” “ACL repair,” “ACL reconstruction,” “ACL operation,” “ACL tear,” “ACL injury,” and “ACL rupture.” The term “ACL surgery” had the highest search volume score across 5 years and was chosen for our search term for article identification.

For internal validity, 2 Google searches were independently performed by 2 reviewers (B.G., T.R.G.) on October 18, 2019, and November 30, 2019. These searches were performed by 2 orthopaedic surgery residents on their personal computers. The reviewers deleted all cookies, caches, and temporary Internet files (including saved passwords, logins, etc) and logged out of all Google accounts before the searches. The results from both searches were identified, and duplicates were removed. The Google search engine was utilized because Google searches comprised 88% to 92% of the online search market share at the time of this study.

Analyses of click-through rates have suggested that approximately ≥70% of “clicks” are for the first 10 search results. A target of the first 50 websites from each search was chosen so as to far exceed that number. In addition, previous PEMAT studies have ranged from targeting the first 10 to 50 websites, so our target was chosen to be consistent with the higher end of previous analyses. The Google search engine rank was averaged from the 2 independently conducted searches.

From the remaining websites, we excluded the following: materials that did not discuss ACL injury or ACL injury treatment, news articles, personal experiences, exclusively audiovisual materials (eg, Vimeo or YouTube videos), reference pages written for health care professionals (eg, Orthobullets or UpToDate), studies from peer-reviewed journals, and advertisements or discussions of a product or service without educational information (eg, instructions on how to contact an office to schedule an appointment or an order form for a KT-1000 arthrometer). Patient education materials that were exclusively audiovisual were excluded because these could not undergo readability analysis. Given the difficulty in the comparison and synthesis of evaluation tools for audiovisual materials and print materials, the focus of this study was placed on print materials and websites.

All search results not meeting exclusion criteria were included. In this way, we included only those web pages that aimed at providing educational materials (instead of news, anecdotes, or nonrelated information) to patients (instead of health care providers or academic professionals).

Qualitative Content Tabulation

All included patient education materials underwent qualitative tabulation of their content including the following categories: discussion of operative management; if operative management specifically described reconstruction, repair, or both; discussion of nonoperative management; advertisement of a physician or group that provided the treatments described; discussion of general background information (anatomy, pathology, prognosis, risk factors); discussion of ACL injury prevention; discussion of work-up or activities related to diagnosis/preoperative management; discussion of postoperative management; and discussion of complications and/or risks of operative management. Online resources were categorized as including advertisements of a medical provider if they stated that a specific institution or group provided the treatments described on the website or a treatment related to ACL injuries within the main text of the educational material.

Readability

The following objective algorithms were utilized to grade readability of the content: Flesch Reading Ease, Flesch-Kincaid Grade Level, Simple Measure of Gobbledygook, Coleman-Liau Index, Gunning Fog Index, and Automated Readability Index (Appendix Table A1). These 6 algorithms have been reliably utilized in previous surgical readability studies. Text unrelated to patient education, including copyright information, references, and links outside of the main text, was excluded from readability analysis.

Table A1

Algorithms Used for Interpretation and Calculation of Readability

Measure	Calculation	Interpretation
Flesch Reading Ease	RE = 206.835 – (1.015 × ASL) – (84.6 × ASW)	90.1-100.0 = 5th-grade material; 70.1-80.0 = 7th-grade material; 50.1-60.0 = 10th- to 12th-grade material; 0.0-30.0 = college graduate material
Flesch-Kincaid Grade Level	0.39 × (total words/total sentences) + 11.8 × (total syllables/total words) – 15.59	Estimates grade level of material
Simple Measure of Gobbledygook	1.0430 × 30 × (words with ≥3 syllables/total words) + 3.1291	Estimates grade level of material
Gunning Fog Index	0.4 × (total words/total sentences) + 100 × (words with ≥3 syllables/total words)	Estimates grade level of material
Automated Readability Index	4.71 × (total characters/total words) + 0.5 × (total words/total sentences) – 21.43	Estimates grade level of material
Coleman-Liau Index	0.0588 (letters per 100 words) − 0.3 (sentences per 100 words) − 15.8	Estimates grade level of material

All utilized open-source readability software (https://www.webfx.com/tools/read-able/check.php). ASL, average sentence length; ASW, average number of syllables per word ; RE, readability ease.

Understandability and Actionability

Using the PEMAT for Printable Materials (PEMAT-P), 2 reviewers (B.G., T.R.G.) individually conducted understandability and actionability analysis for the included materials. The PEMAT yields separate understandability and actionability scores on a 0% to 100% scale for each educational material evaluated. A higher score represents a better level of understandability or actionability, with a score of ≥70% as the standard for the resource to be considered to have adequate understandability and actionability.

During PEMAT analysis, 2 resources were excluded because they were removed from the Internet during the time of the study (Appendix Table A2). PEMAT analysis was completed after additional analyses were performed; therefore, these 2 patient education materials were included in qualitative content tabulation and readability analyses.

Table A2

Included Websites (n = 39)

Websites
https://rothmanortho.com/specialties/treatments/acl-reconstruction-surgery
https://uvahealth.com/services/sports-medicine/acl-surgery
https://www.eliteorthopaedic.com/acl-reconstruction a
https://www.franciscanhealth.org/health-care-services/acl-surgery-599
https://www.hss.edu/conditions_acl-reconstruction-new-advances.asp
https://www.ortho.wustl.edu/content/Patient-Care/3181/Services/Pediatric-and-Adolescent-Orthopedic-Surgery/Overview/Knee-Education-Overview/ACL.aspx
http://www.ossmd.com/acl-reconstruction
https://www.stlouischildrens.org/conditions-treatments/orthopedics/teaching-tools/acl-reconstruction
https://www.ucsfhealth.org/education/after-acl-surgery
https://www.upmc.com/services/sports-medicine/services/acl-program/treatment/surgical/before-surgery
https://www.webmd.com/pain-magement/knee-pain/acl-surgery-what-to-expect#1
https://healthcare.utah.edu/orthopaedics/specialties/acl-reconstruction-prevention.php
https://kidshealth.org/en/teens/acl-surgery.html
https://medlineplus.gov/ency/article/007208.htm
https://share.upmc.com/2015/04/recovery-time-for-acl-reconstruction-surgery
https://smgortho.com/procedure/acl-reconstruction
https://treasurevalleyhospital.com/newsroom/ArticleID/17/Evaluating-Costs-for-Your-ACL-Surgery-What-You-Need-to-Know
https://wexnermedical.osu.edu/sports-medicine/injuries/knee/acl-reconstruction
https://www.childrenscolorado.org/conditions-and-advice/conditions-and-symptoms/conditions/acl-injury
https://www.healthgrades.com/right-care/acl-surgery/anterior-cruciate-ligament-acl-surgery
https://www.healthline.com/health/acl-reconstruction
https://www.healthlinkbc.ca/health-topics/hw28289
https://www.hommenorthopedics.com/anterior-cruciate-ligament-acl-reconstruction.html
https://www.hss.edu/condition-list_torn-acl.asp
https://www.mayoclinic.org/tests-procedures/acl-reconstruction/about/pac-20384598
https://www.medicinenet.com/torn_acl/article.htm
https://www.medstarortho.org/treatments/acl-surgery
https://www.medstarwashington.org/our-services/orthopaedics/treatments/acl-repair-surgery
https://www.mottchildren.org/health-library/hw28289
https://www.nwh.org/surgery/surgical-discharge-instructions/acl-reconstruction-instructions
https://www.oamortho.com/acl-reconstruction-orthopedic-associates-meadville-pc.html
https://www.orthonebraska.com/treatment/acl-reconstruction-surgery
https://www.orthopedicandfracturespecialists.com/post-op-acl
https://www.pennmedicine.org/for-patients-and-visitors/find-a-program-or-service/orthopaedics/where-is-your-pain/knee/knee-surgery/knee-ligament-repair/acl-surgery a
https://www.performancehealth.com/articles/before-and-after-your-acl-surgery-what-to-do-and-what-you-ll-need
https://www.ucsfhealth.org/education/acl-reconstruction-surgery
https://www.verywellhealth.com/acl-surgery-making-a-decision-2548473
https://www.virginiamason.org/acl-surgery
https://www.webmd.com/pain-magement/knee-pain/acl-surgery-what-to-expect

These resources were excluded during Patient Education Materials Assessment Tool (PEMAT) analysis because the content was removed from the Internet during the time of the study. PEMAT analysis was completed after additional analyses were performed; therefore, these patient education materials were included in qualitative content tabulation and readability analyses.

Statistical Analysis

Interrater reliability of the PEMAT-P was calculated using the Cohen kappa statistic. The magnitude of the kappa value was interpreted using the criteria set by Landis and Koch because these criteria were used by PEMAT developers and later evaluators to measure the reliability of PEMAT scoring. The Spearman rank correlation coefficient (ρS) was used to assess the association between search engine rank, readability, understandability, and actionability. Stata 16 (StataCorp) was utilized for statistical calculations. For all analyses, statistical significance was defined as P < .05.

Results

Search Results and Qualitative Content Tabulation

After the removal of duplicate websites, a total of 68 unique websites were identified from the 2 independently conducted searches, and 39 websites met inclusion criteria (Appendix Table A2). Of the 29 excluded websites, 9 were excluded because their focus was the advertisement of a service or product, 8 were excluded because they were references for medical trainees or professionals, 6 were excluded because they were audiovisual materials, 4 were excluded because they were news reports, and 2 were excluded because they were peer-reviewed journal articles. There was no significant difference in the mean Google search rank between the excluded materials (29.69 ± 14.16 [range, 4-49]) and the included materials (23.94 ± 14.90 [range, 1-50]) (P = .114).

Of the 39 included websites, 27 discussed general background information (anatomy, pathology, prognosis, risk factors), 4 discussed ACL injury prevention, 11 discussed nonoperative management, 17 discussed work-up or activities related to diagnosis/preoperative management, 35 discussed operative management, 28 discussed postoperative management, 8 discussed complications and/or risks of operative management, and 21 advertised a physician or group that provided the treatments described. Of the 36 websites that discussed operative management, all 35 described technical aspects of ACL reconstruction. The association between content categories and Google rank was determined, and websites discussing injury prevention and those advertising a physician or group providing ACL treatment were noted to be associated with significantly worse (lower) Google ranking (injury prevention: ρS = 0.41 [P = .014]; advertising: ρS = 0.35 [P = .042]) (Table 1).

Table 1

Results of Qualitative Content Tabulation

Content	No. of Patient Education Materials Reporting Content	Association With Google Rank (ρS)	P
Background information	27	–0.02	.926
Injury prevention	4	0.41	.014
Nonoperative management	11	0.17	.318
Work-up or activities related to diagnosis/preoperative management	17	–0.12	.476
Operative management	35	–0.08	.633
Postoperative management	28	0.13	.473
Complications and/or risks of operative management	8	–0.33	.055
Advertised a physician or group that provided the treatments described	21	0.35	.042

Bolded P values indicate a statistically significant association with Google rank (P < .05).

Across all readability measures reporting grade levels, the mean grade level ranged from 10.01 to 13.09 (ie, first year of college) (Figure 1 and Appendix Table A3). The number of websites rated at having a readability score below the 6th-grade level was 4 according to the Automated Readability Index, 1 according to the Simple Measure of Gobbledygook, and 1 according to the Flesch-Kincaid Grade Level. No readability measures were statistically significantly associated with Google rank (Appendix Table A3).

Figure 1.

Overall mean readability scores of the 39 included patient education websites. The horizontal line indicates the American Medical Association (AMA) and National Institutes of Health (NIH) reading grade recommendation for medical education materials designed for the public. SMOG, Simple Measure of Gobbledygook.

Table A3

Results of Readability Analysis

Variable	Mean ± SD	Association With Google Rank (ρS)	P
Flesch Kincaid Grade Level	10.01 ± 2.36 (5.1-15.7)	0.26	0.104
SMOG Readability Formula	10.02 ± 2.14 (5.5-16.1)	0.23	0.162
Gunning Fog Index	13.09 ± 2.60 (7.5-18.2)	0.26	0.107
Automated Readability Index	10.02 ± 2.14 (5.2-17.9)	0.27	0.103
Coleman-Liau Index	12.46 ± 1.53 (9.2-16.1)	0.18	0.272

PEMAT-P Scores

Interrater reliability demonstrated moderate interrater agreement (kappa = 0.50 ± 0.04; P < .001). Mean understandability and actionability scores were 59.18% ± 10.86% (range, 27.27-87.50) and 34.41% ± 22.31% (range, 0.00-83.33), respectively (Table 2). Only 5 (12.82%) and 1 (2.56%) resource scored above the 70% adequate PEMAT-P threshold mark for understandability and actionability, respectively. Understandability and actionability were not significantly associated with Google rank (understandability [n = 37]: ρS = –0.13 [P = .441]; actionability [n = 37]: ρS = 0.28 [P = .096]).

Table 2

Patient Education Materials Assessment Tool for Printable Materials Scores

	Reviewer 1		Reviewer 2
	No. of Studies Qualified for Analysis	Percentage of Qualified Studies Fulfilling Criteria	No. of Studies Qualified for Analysis	Percentage of Qualified Studies Fulfilling Criteria
Understandability
The material makes its purpose completely evident.	37	54.05	37	59.46
The material does not include information or content that distracts from its purpose.	37	37.84	37	35.14
The material uses common, everyday language.	37	54.05	37	75.68
Medical terms are used only to familiarize audience with the terms. When used, medical terms are defined.	37	78.38	37	72.97
The material uses the active voice.	37	27.03	37	72.97
Numbers appearing in the material are clear and easy to understand. (Excluded if no numbers)	28	67.86	22	90.91
The material does not expect the user to perform calculations.	37	81.08	37	100.00
The material breaks or “chunks” information into short sections.	36	97.22	36	94.44
The material’s sections have informative headers. (Excluded if qualified as very short material)	35	82.86	36	91.67
The material presents information in a logical sequence.	37	78.38	37	86.49
The material provides a summary. (Excluded if qualified as very short material)	34	8.82	36	5.56
The material uses visual cues to draw attention to key points. (Videos excluded)	37	40.54	37	16.22
The material uses visual aids whenever they could make content more easily understood.	37	43.24	37	24.32
The material’s visual aids reinforce rather than distract from the content. (Excluded if no visual aids)	17	35.29	12	50.00
The material’s visual aids have clear titles or captions. (Excluded if no visual aids)	17	47.06	12	58.33
The material uses illustrations and photographs that are clear and uncluttered. (Excluded if no visual aids)	17	47.06	12	66.67
The material uses simple tables with short and clear row and column headings. (Excluded if no visual aids)	1	100.00	1	100.00
Overall understandability (n = 37), mean ± SD (range), %		56.15 ± 13.73(30.77-83.33)		62.21 ± 13.29(27.27-87.50)
Actionability
The material clearly identifies at least one action the user can take.	37	64.86	37	78.38
The material addresses the user directly when describing actions.	37	54.05	37	72.97
The material breaks down any action into manageable, explicit steps.	37	27.03	37	37.84
The material provides a tangible tool (eg, menu planners, checklists) whenever it could help the user take action.	37	0.00	37	0.00
The material provides simple instructions or examples of how to perform calculations. (Excluded if no calculations)	0	0.00	0	0.00
The material explains how to use the charts, graphs, tables, or diagrams to take action. (Excluded if no charts, graphs, tables, or diagrams)	1	100.00	2	0.00
The material uses visual aids whenever they could make it easier to act on the instructions.	37	5.41	37	2.70
Overall actionability (n = 37), mean ± SD (range), %		30.90 ± 26.67(0.00-83.33)		37.93 ± 23.09(0.00-80.00)

Understandability was positively correlated with the Flesch Reading Ease (ρS = 0.37; P = .025) but otherwise not significantly associated with other readability measures (Table 3). Actionability was negatively associated with 2 readability measures (Flesch-Kincaid Grade Level and Gunning Fog Index; P < .10).

Table 3

Association Between Patient Education Materials Assessment Tool for Printable Materials and Readability Scores

	Overall Understandability		Overall Actionability
	ρS	P	ρS	P
Flesch Reading Ease	0.369	.025	0.317	.056
Flesch-Kincaid Grade Level	–0.287	.085	–0.194	.250
Simple Measure of Gobbledygook	–0.294	.077	–0.212	.208
Gunning Fog Index	–0.340	.040	–0.209	.215
Automated Readability Index	–0.204	.226	–0.120	.480
Coleman-Liau Index	–0.199	.237	–0.263	.115

Discussion

The study findings revealed that ACL surgery–related publicly available online materials have poor readability, understandability, and actionability as well as limited discussion of relevant information regarding surgical decision-making. First, our content analysis suggested that key information related to surgical decision-making, such as complications and risks, is not widely discussed. Second, 6 different readability algorithms demonstrated that the syntax and vocabulary used created a reading difficulty far exceeding the ability of average adults in the United States. Lastly, materials scored poorly in understandability and actionability, suggesting that patients are unable to comprehend the information presented or easily identify available actions.

A majority of online patient education materials were found to lack content that could serve to empower patients to engage with their health care provider in decision-making. Of 7 aspects of information related to surgical decision-making (general background information, work-up related to diagnosis, injury prevention, surgical procedure details, postoperative management, nonoperative management, and complications/risks of surgery), only 3 of these categories (background information, surgical procedure details, and postoperative management) were discussed in more than half of materials. Only 28% of materials discussed nonoperative management strategies, and 21% discussed the complications/risks of surgery. With 90% of materials discussing the details of operative management and only approximately a quarter of materials discussing operative risks and alternative nonoperative management, materials may bias patients toward pursuing surgery by not fully describing all aspects related to surgical decision-making.

The discussion of alternative therapies and the risks and complications of surgery has been deemed by the American College of Surgeons as a key aspect to informed consent and ethical surgical practice. These findings are consistent with qualitative analyses in other studies. Duncan et al qualitatively evaluated websites that appeared under the search query “ACL reconstruction” and found that while 62.5% of websites described the surgical technique of ACL reconstruction, only 30% discussed surgical eligibility/alternative therapies and only 30% discussed complications. Sambandam et al qualitatively evaluated online resources related to knee arthroscopic surgery and demonstrated that only 1.5% to 12% of websites adequately reported on all major pieces of information associated with surgical decision-making (diagnosis, procedure details, alternative treatments, postoperative management, complications, and long-term prognosis). Our qualitative content tabulation also revealed that over half of all patient education materials advertised a physician or group. Currently, the appropriateness of this advertising is controversial among physician organizations and is beyond the scope of this study.

The AMA and NIH currently recommend that public health literature be written at the 6th-grade reading level or lower to be understood by the average adult. Our readability analysis demonstrated that the mean readability grade level ranged from the 10th grade to a level appropriate for college freshmen (13th grade). The high level of literacy required to read ACL surgery educational materials is troubling, as it is estimated that 80 to 90 million adults in the United States have limited health literacy. Furthermore, deficits in health literacy have been correlated to poor treatment adherence and worse outcomes including higher rates of rehospitalization and higher mortality. Unfortunately, poor readability is not unique to ACL surgery literature. The readability scores demonstrated in this study are consistent with readability scores reported in previous studies examining online orthopaedic patient education materials involving hand surgery (average, 11.92-grade level), meniscal tears (average, 11.14-grade level), hip arthroscopic surgery (average, 12.97-grade level), and orthopaedic trauma (average, 8.8-grade level). Akinleye et al compared the readability scores of educational materials on 5 different arthroscopically treated abnormalities (ACL tear, meniscal tear, hip labral tear, shoulder labral tear, and rotator cuff tear) and found that those on ACL tear had the highest average reading grade level at 10.73. This suggests that while poor readability is prevalent throughout orthopaedic materials, ACL tear–related websites may need more improvement than websites of other orthopaedic injuries to reach AMA and NIH recommendations.

ACL reconstruction educational materials also scored poorly on assessments of both understandability and actionability. The developers of the PEMAT set a minimum threshold of 70% for a text to be considered understandable and actionable by readers ; however, only 5 websites (12.82%) met this threshold for understandability, and only 1 website (2.56%) did so for actionability. Additionally, the online ACL surgery resources only fulfilled a mean of approximately 59% of understandability items and 34% of actionability items. These poor understandability scores are similar to those reported in other surgical specialties including otolaryngology, urology, and neurosurgery. These actionability scores, however, rank below scores reported in other fields. Materials related to laryngectomy and prostate biopsy as well as Medline-derived patient education materials all were criticized for achieving actionability scores of only roughly 60%.

The poor understandability and actionability scores found in this study are consistent with those of previous studies regarding the quality of online orthopaedic materials, which have used alternative tools such as Health On the Net Code of Conduct (HONcode) certification, Journal of the American Medical Association (JAMA) benchmarks, and DISCERN criteria. HONcode certification is an accreditation given to websites by a Swiss nongovernmental organization, which deems a website to satisfy 8 criteria that focus on the accuracy and reliability of published information. JAMA benchmarks are used to evaluate resource quality on the basis of 4 criteria, which are similar to HONcode criteria, with an emphasis on disclosure and authorship transparency. DISCERN criteria are used to grade quality on the basis of 16 questions, which focus on the completeness of content described. Bruce-Brand et al analyzed 45 websites related to ACL reconstruction and found that the mean DISCERN score was 41.1, or barely categorized as “fair” quality (39-50 points). Furthermore, 42.2% of websites were found to be “poor” quality (27-38 points). Only 17.8% of websites were HONcode certified, and only 22.2% of websites fulfilled all 4 JAMA benchmark criteria. Cassidy and Baker conducted a systematic review of 38 orthopaedic surgery studies using either ≥1 of these criteria or nonstandardized qualitative analyses and found that the majority of resources either did not have HONcode accreditation or scored poorly via JAMA and DISCERN criteria. None of the analyzed studies found that >50% of websites demonstrated scores consistent with high-quality content.

Our study analyzed the association between the various measures and ranking calculated in this study. Google rank was associated with content but not with readability, understandability, or actionability. Specifically, the presence of advertisements was associated with worse (later) Google ranking (Table 1). While this study cannot prove the reasons for these changes in rank, one of Google’s publicly known algorithms is PageRank, which attempts to measure the importance of a target website by calculating the number of links in other nontarget websites to the target website. Thus, it is not surprising that websites that advertise for a specific physician or group would not be valuable to other websites and thus not receive higher Google ranking. The lack of association between readability, understandability, and actionability and Google rank may be because of most resources scoring poorly in these areas.

Limitations

This study had several limitations. First, we analyzed online print-based materials, but patients may also utilize audiovisual resources online. Second, this study used the search term “ACL surgery” but did not seek to expand to other related or more specific search terms such as “complications.” While the choice to do this was supported using the Google Trends feature of Google.com as described in the Methods section, this may have skewed our analysis away from resources that did not focus on specific areas of ACL surgery such as complications.

Third, this study utilized the first 50 search results from the Google search engine. While the Google search engine carried approximately 88% to 92% of the online search market share at the time of this study, other search engines could theoretically provide different websites from the Google search engine. Additionally, the top 50 results within a Google search could be different at various time points and search locations. This was seen in our results: between 2 reviewers who completed their searches 6 weeks apart, 68 unique websites were found. While it is possible that new websites were created between these time points, it is also likely that different results were found because of Google’s proprietary search algorithm, which has an element of personalization. Publicly, it is known that Google uses location, search history via browser cookies, and Google account information to personalize search results. In a rigorous analysis of 200 Google user accounts, a study found that an average of 11.7% of results were different across users because of personalization features and that most of the differences due to personalization occurred in later ranked web pages. This same study found that these results were driven mostly by Google account information and Internet Protocol (IP) address (ie, geolocation). As such, our reviewers attempted to reduce the bias associated with personalization by logging out of Google accounts and clearing all temporary Internet files including caches and cookies. Our study was unable, however, to control for the effects of our IP addresses and other proprietary data, which are collected but not publicly confirmed or acknowledged by Google. Additionally, we were unable to predict what cookies and private information that target patients may have stored on their computer at a specific search time point. The results of this study then are inherently difficult to reproduce and cannot be interpreted as an analysis of all available online resources. Rather, these results should be interpreted as a sampling of available resources to patients and the possible quality of materials that patients are likely to encounter. In this endeavor, we sought to capture the typically seen web pages by patients by extending our search to 50 results, which exceeds the first 10 search results that are most commonly chosen.

Fourth, this study is not able to comment on the level of benefit that patients may experience from these online educational materials. This study did not ask patients to read these resources directly and provide patient-reported scores regarding the usefulness of the online materials. It is possible that regardless of how poorly written an educational material is, a patient may still benefit from it. In this regard, our study is only able to suggest that online resources have areas for improvement but not able to suggest that patients do not benefit from them. Knowledge of this might assist in determining if lacking certain content categories (as tabulated in our qualitative analysis) causes actual detriment to patients.

Fifth, the PEMAT requires grading in a subjective manner, and implicit bias could not be fully eliminated. To limit this bias, 2 reviewers independently performed the assessment, which demonstrated agreement with moderate interrater reliability that is consistent with previous studies. While the PEMAT has been validated, to our knowledge, no other PEMAT studies that have evaluated orthopaedic abnormalities exist. Future research should utilize the PEMAT to evaluate other orthopaedic injuries. While orthopaedic studies that have used JAMA, HONcode, and DISCERN criteria exist, none of these has focused on the reader’s ability to understand or act on the material. Instead, they have focused on the completeness of content (DISCERN), the reliability of content (HONcode), and the transparency of authorship and sources (JAMA). Additionally, JAMA benchmarks are difficult to interpret; higher scores are better, but it is not clear what score threshold is acceptable. HONcode certification is also poorly regulated, and the HONcode certification logo may be falsely displayed on a web page without being truly certified. Also, future research should be conducted to include the accuracy of educational materials, which neither readability nor the PEMAT takes into account.

In addition to further evaluations of online orthopaedic materials, quality improvement studies should be conducted on handouts and educational materials provided by surgeons themselves. Although, as discussed in our Limitations section, the results of Google searches may vary, the data analyzed in this study suggest that the quality of publicly searchable online education materials may be poor and patients may thus need to lean on educational materials given by surgeons. Additionally, future research should seek to look at commonly available audiovisual materials.

Conclusion

Qualitative content tabulation as well as readability, understandability, and actionability analyses revealed several areas for improvement in online ACL surgery–related patient education materials. First, readability analysis suggested that current materials use words and sentences that are too complex to be understood by patients. Second, PEMAT grading suggested that readers can neither process key messages within the text (poor understandability) nor identify actions that they can take based on messages within the material (poor actionability). Nearly all resources could improve their scores by providing summaries, adding visual cues to draw attention to key points, adding tangible tools to assist readers in taking action, and breaking down actions into explicit steps. Finally, qualitative content tabulation revealed that most materials lack discussions of the risks of operative management and the alternatives to operative procedures described.