Online Resources for Rotator Cuff Repair: What are Patients Reading?

PURPOSE: The purpose of this study was to use a novel scoring system to evaluate the content and grade the quality of websites that patients may use to learn about rotator cuff repair.
METHODS: Two search terms ("rotator cuff repair" and "rotator cuff surgery") were entered into 3 Internet search engines (Google, Yahoo, and Bing). We scored the quality of information using a novel scoring system. Website quality was further assessed by the Journal of the American Medical Association (JAMA) benchmark criteria and Health on the Net Foundation (HON) code certification. The readability of the websites was evaluated with the Flesch-Kincaid score.
RESULTS: We evaluated 47 websites. The average quality for all websites was 6.47 ± 5.21 (maximum 20 points). There was a large difference in scores between the top 5 websites and the remaining websites (16.30 vs 5.51, P < .001). There was no difference in scores when comparing the 3 different search engines (P = .85). The mean reading level was 10.17 ± 2.24. Reading level was not significantly correlated with quality (rs = 0.14, P = .36). The average JAMA benchmark criteria score for all websites was 2.34 ± 1.11 (maximum 4 points). JAMA criteria score was not significantly correlated with quality (rs = 0.02, P = .91). Sites without HONcode certification had higher quality scores (8.33 ± 4.80) than sites with HONcode certification (6.18 ± 4.66), but this difference was not statistically significant (P = .15).
CONCLUSION: The quality of patient-level information on rotator cuff repair on the Internet is both incomplete and written at a reading level higher than current recommendations. Information quality is not significantly correlated with reading level or JAMA criteria, and does not depend on the search term used or HONcode certification. CLINICAL RELEVANCE: Patients having rotator cuff repair may seek information on the Internet; the information may require surgeon clarification.

Rotator cuff disorders are one of the most common causes of disability related to the shoulder, and there has been a dramatic increase in national rates of rotator cuff repair. Repairs of rotator cuff tears show excellent outcomes, with statistically significant improvements in most patients.3, 4, 5, 6, 7, 8, 9, 10, 11 However, from the patient's perspective, the prospect of undergoing a rotator cuff repair can be daunting, from understanding how best to prepare for surgery and undergoing anesthesia to optimal strategies for a successful recovery. To learn about their condition and prepare for surgery, many patients will turn to the Internet as a source of information. Today, it is estimated that 9 of 10 U.S. adults use the Internet, and 72% of Internet users report having searched for a health related topic online. Furthermore, the Internet has been shown to be the preferred source of health information for seniors, a demographic that experiences a very high rate of rotator cuff tears.14, 15 Physicians believe that the Internet can lead to patients becoming better informed. However, many fear that Internet health misinformation and misinterpretation can lead to unrealistic patient expectations, which can damage the patient-physician relationship16, 17 and impact patient satisfaction and patient-reported outcomes.18, 19, 20, 21 Henn et al. studied the correlation of preoperative expectations and postoperative outcomes scores in rotator cuff repair and found that higher preoperative expectations of surgical success was associated with higher postoperative performance on self-assessed outcome scores.

Previous studies have evaluated the quality of online information available to patients undergoing orthopaedic procedures.23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35 Given the prevalence of Internet users and searches for health information, as well as the high number of rotator cuff repairs performed each year, it is likely that patients scheduled for rotator cuff repair are likely to search the Internet to learn about the surgery. Thus, it is crucial to evaluate the quality of the online resources for rotator cuff repair.

The purpose of this study was to use a novel scoring system to evaluate the content and grade the quality of websites that patients may use to learn about rotator cuff repair. It was hypothesized that most website quality will be incomplete and not cover the breadth of relevant information related to rotator cuff repair.

Methods

Websites were selected by using a combination of search terms that patients would be most likely to use to find information about the procedure. These search terms were “rotator cuff repair” and “rotator cuff surgery.” Both terms were entered into 3 separate popular search engines (Google, Bing, Yahoo), which resulted in 6 separate searches (2 search terms × 3 search engines). Websites were recorded from the top 25 results from each search, for a total of 150 results. The search was completed in September 2018.

Websites were grouped by search term and by search engine. When a duplicate website was identified, the term and search engine was selected based on which combination yielded the top-ranked (i.e., earliest) website for a search. Peer-reviewed primary literature was excluded because it was assumed that these would likely exceed comprehension for most patients and also that most patients would not have access to scientific literature. Video content and content for medical professionals were also excluded because of their intended target audience.

After establishing a final list of included websites that discussed rotator cuff repair, the websites were read and independently scored by 4 different evaluators (B.G., W.S., T.D., L.L.), 3 of whom are sports medicine fellowship–trained orthopaedic surgeons. To score the quality of the websites, an author-derived score with a maximum of 20 points was created. The components of the scoring system were agreed on by all authors who are experienced orthopaedic surgeons and consisted of important aspects through each stage of undergoing a rotator cuff repair. The scoring system consisted of elements of diagnosis, treatment options, surgical factors, and postoperative instructions. Each criterion was assigned a 1-point value. Scores for each of the 4 reviewers were combined and averaged to generate a mean quality score for each website included. The scoring card is shown in Table 1.

Table 1

Scoring System to Assess Website Information on Rotator Cuff Repair and Rotator Cuff Surgery

Scoring Criteria
Preoperative
Signs and symptoms of tear
Nonoperative options
Surgical options for non-reparable tears
Optimal surgical preparation strategies
Timing of surgery
Surgical
Indications
Length
Surgical option: open or mini-open
Surgical option: arthroscopic
Anesthesia
Risks and complications
Outcome (recovery time frame)
Postoperative
Effect of general health conditions on recovery
Duration of hospital stay
Immediate post-op instructions (sling, ice, rest, etc.)
Pain control
Physical therapy instructions
Time frame with return to activities of daily living
Indications on when to consult your surgeon

Readability of each website was assessed using the Flesch-Kincaid (FK) method, which is measured by Microsoft Word (Microsoft, Redmond, WA). The FK method is a type of readability test that indicates how difficult a passage is to understand in English. It assigns a grade level that corresponds to the highest level of education the reader must have to comprehend the reading. Readability of websites was compared to a fifth-grade standard, a recommendation recently proposed by the Joint Commission.

We used The Journal of the American Medical Association (JAMA) benchmark criteria to assess each website in 4 possible areas: (1) authorship; (2) listing of sources used; (3) disclosure of ownership, sponsorship, funding, conflict of interest; and (4) date of update. One point was awarded for each criteria, making a total of 4 points.

The Health on the Net Foundation (HON), a nonprofit organization accredited to the United Nations, certifies websites that meet their quality and reliability standards. Each website was assessed for the presence of HONcode certification.

Statistical Analysis

Statistical analysis was performed using STATA statistical software (Version 14.2, Statacorp, College Station, TX). Independent sample t-tests for normally distributed data and Mann-Whitney U test for non-normally distributed data were used to determine differences between quality score and search term and HONcode certification. Correlation analyses (Spearman rank correlation coefficient for non-normally distributed data) was used to determine association between quality and FK scores and JAMA score. Intrarater and inter-rater reliability for the quality scores were assessed with the single-measure, 2-way random-effects, absolute agreement definition of the intraclass correlation coefficient. P < .05 was used to determine statistical significance. No funding was required for this study.

Results

Using 2 separate search terms, “rotator cuff repair” and “rotator cuff surgery,” the initial search yielded 150 results. Seventy-five websites were removed as duplicates, leaving 75 websites to evaluate (Fig 1). Twenty-eight websites were excluded because they were video content (14 sites), peer-reviewed primary literature (9 sites), a website intended for medical professionals (3 sites), broken link (1 site), and selling a product (1 site). After articles were excluded, there were 47 websites left for review (see Appendix 1 for individual website itemized scores). Intrarater reliability was 99.2% (95% CI: 98.6% to 99.6%). The inter-rater reliability among all 4 raters was 95.7% (95% CI: 92.3% to 97.6%).

Fig 1

Flow diagram of website selection.

Appendix 1

Website URLs With Score Breakdown Listed in Order of Highest Score to Lowest Score

URL	Pre-Score	Sx-Score	Post-Score	Total score
http://www.orthop.washington.edu/?q=patient-care/articles/sports/arthroscopic-shoulder-surgery-for-the-treatment-of-rotator-cuff-tears	4	7	6.25	17.25
https://www.healthgrades.com/procedures/rotator-cuff-surgery	2.75	6	7	15.75
https://drmillett.com/double-row-arthroscopic-rotator-cuff-repair-for-rotator-cuff-tears/	3.75	5	7	15.75
https://orthoinfo.aaos.org/en/treatment/rotator-cuff-tears-surgical-treatment-options/	3	5.75	6.25	15
https://www.healthline.com/health/rotator-cuff-repair	3.5	5.5	5	14
https://www.hopkinsmedicine.org/healthlibrary/test_procedures/orthopaedic/rotator_cuff_repair_92,P07682	3.25	5.25	5.25	13.75
https://medlineplus.gov/ency/article/007207.htm	2.75	5.75	5.25	13.75
https://www.sports-health.com/sports-injuries/shoulder-injuries/rotator-cuff-repair-surgery	3	4.75	5.5	13.25
http://www.orthosports.com.au/content_common/pg-open-rotator-cuff-repair.seo	2	5	6.25	13.25
https://www.webmd.com/pain-management/rotator-cuff-surgery	2.75	5	5	12.75
https://www.medigo.com/en-us/procedure-pages/orthopedics/rotator-cuff-surgery	1.75	6.75	2.25	10.75
https://newsok.com/article/3882852/whats-it-like-to-have-your-rotator-cuff-repaired	1.25	4.25	5	10.5
https://www.howardluksmd.com/education/common-injuries/recovery-from-rotator-cuff-surgery/	1.5	3.25	5.5	10.25
https://www.g2orthopedics.com/rotator-cuff-surgery-faqs/	1.5	3	5.75	10.25
http://www.kneeandshouldersurgery.com/shoulder-disorders/rotator-cuff-disorders/rotator-cuff-tears/	3.25	3.75	2.75	9.75
https://sportssurgerychicago.com/rotator-cuff-repair-protocol-treatment-westchester-oakbrook-hinsdale-il/	2	3	4.25	9.25
https://www.shoulderdoc.co.uk/article/62	2	3.5	3.5	9
https://www.massgeneral.org/ortho/services/sports/rehab/Rotator%20cuff%20repair%20rehabilitation%20protocol.pdf	2	1.25	5.5	8.75
https://www.nordorthopaedics.com/en/shoulder-surgery/rotatory-cuff-tear-repair	1.25	3.75	3.25	8.25
https://en.wikipedia.org/wiki/Rotator_cuff	2.25	3	2.75	8
https://www.rothmaninstitute.com/specialties/treatments/rotator-cuff-surgery	2.5	3.5	2	8
https://myhealth.alberta.ca/Health/aftercareinformation/pages/conditions.aspx?hwid=ug3498	0.75	1.75	5	7.5
https://www.verywellhealth.com/rotator-cuff-tear-surgery-2549776	2.75	2.75	1	6.5
https://my.clevelandclinic.org/health/diseases/8291-rotator-cuff-tears-overview	1.5	1.5	3.5	6.5
https://www.arksurgicalhospital.com/5-tips-recovering-rotator-cuff-surgery-returning-game/	0.75	1.25	4	6
https://www.mayoclinic.org/diseases-conditions/rotator-cuff-injury/diagnosis-treatment/drc-20350231	3	2.75	0	5.75
https://www.webmd.com/pain-management/rotator-cuff-tear	1.75	1.5	2.25	5.5
https://www.verywellhealth.com/rehab-after-rotator-cuff-surgery-2549905	0	0.25	4.75	5
https://www.hopkinsmedicine.org/orthopaedic-surgery/specialty-areas/shoulder/treatments-procedures/failed-rotator-cuff-repairs.html	1	1.5	2	4.5
https://www.anationinmotion.org/value/rotator-cuff/	1.5	2.25	0.75	4.5
https://www.premierortho.com/physical-therapy/long-recovery-period-shoulder-surgery/	0	1.25	3	4.25
https://healthfully.com/average-following-rotator-cuff-surgery-5418081.html	1	0.25	2.5	3.75
https://kneeandshoulderclinic.com.au/shoulders/surgical-conditions/rotator-cuff-repair/	2.75	0.75	0.25	3.75
https://www.medstargeorgetown.org/our-services/orthopaedics/conditions/shoulder-conditions/rotator-cuff-surgery/	2	1.75	0	3.75
https://truesportsphysicaltherapy.com/what-can-i-expect-after-rotator-cuff-surgery/	0.25	0.5	3	3.75
http://www.stoneclinic.com/rotator-cuff-repair-rehab-protocol/	0.5	0	3.25	3.75
https://www.regenexx.com/rotator-cuff-tear-healing-without-surgery/	2	1.25	0	3.25
https://orthoinfo.aaos.org/en/diseases--conditions/rotator-cuff-tears/	3	0.25	0	3.25
https://www.howardluksmd.com/orthopedic-social-media/do-i-need-rotator-cuff-tear-surgery/	2	1.25	0	3.25
https://www.sports-health.com/treatment/shoulder-injury-treatment/different-types-rotator-cuff-surgery	0.25	2	0	2.25
https://www.mayoclinic.org/diseases-conditions/rotator-cuff-injury/symptoms-causes/syc-20350225	2.25	0	0	2.25
https://www.healthline.com/health/rotator-cuff-injury-stretches	2	0	0	2
https://www.webmd.com/pain-management/what-is-my-rotator-cuff	1.75	0	0	1.75
https://medlineplus.gov/ency/patientinstructions/000357.htm	1	0	0.75	1.75
https://www.mocnyc.com/arthroscopic-rotator-cuff-repair/	0	1.75	0	1.75
https://medlineplus.gov/ency/presentations/100229_1.htm	0	0	0	0

Overall, the included websites had an average score of 6.47 ± 5.21 of 20 total points. By subsection, this included 1.63 ± 1.25 (32.6%) of 5 total points for the presurgical subsection, 2.27 ± 2.17 (32.4%) of 7 total points for the surgical subsection and 2.58 ± 2.45 (32.2%) of 8 total points for the postoperative subsection. The 3 questions with the highest percentage representation from all articles were “signs and symptoms of a tear (60%),” “physical therapy (54%),” and “surgical options: arthroscopic (51%).” The lowest criteria were “effect of general health conditions on recovery (12%),” “surgical options for non-reparable tears (13%),” and “when to consult your surgeon (13%)” (Table 2). The 5 highest scored websites can be found in Table 3. There was a large difference in scores between the top 5 websites and the remaining websites (16.30 vs 5.51; P < .001) (Table 3). There was no difference in scores when comparing the 2 different search terms (P = .85).

Table 2

Average Website Score by Topic

	Percent of Websites Including Topic
Pre Surgical
Signs and symptoms of tear	60∗
Nonoperative options	42
Surgical options for non-reparable tears	13†
Optimal surgical preparation strategies	21
Timing of surgery	26
Surgical
Indications	44
Length	14
Surgical option: open or mini-open	42
Surgical option: arthroscopic	51∗
Anesthesia	18
Risks and complications	24
Outcome (recovery time frame)	34
Postoperative
Effect of general health conditions on recovery	12†
Duration of hospital stay	17
Immediate post-op instructions (sling, ice, rest, etc.)	48
Pain control	36
Post-op restrictions	40
Physical therapy instructions	54∗
Time frame with return to activities of daily living	39
Indications on when to consult your surgeon	13†

Top 3 represented by all websites.

Bottom 3 represented by all websites.

Table 3

Summary of Quality of Results

Indicator	Mean Quality (Maximum, 20)	P Value
All websites	6.47 ± 5.21
Top 5 websites	16.30	<.001
http://www.orthop.washington.edu/?q=patient-care/articles/shoulder/repair-of-rotator-cuff-tears.html
http://www.orthop.washington.edu/?q=patient-care/articles/sports/arthroscopic-shoulder-surgery-for-the-treatment-of-rotator-cuff-tears
https://www.healthgrades.com/procedures/rotator-cuff-surgery
https://drmillett.com/double-row-arthroscopic-rotator-cuff-repair-for-rotator-cuff-tears/
https://orthoinfo.aaos.org/en/treatment/rotator-cuff-tears-surgical-treatment-options/
All remaining websites	5.51
Search term: rotator cuff repair	9.03 ± 5.59	.85
Search term: rotator cuff surgery	6.65 ± 3.90
FK grade level	Not significantly correlated with quality
JAMA benchmark criteria	Not significantly correlated with quality
No HONcode certification	8.33 ± 4.80	.15
HONcode certification	6.18 ± 4.66

FK, Flesch-Kincaid; JAMA, Journal of the American Medical Association.

The average FK grade for all websites was 10.17 ± 2.24 (range 5.8-16.4). There was only 1 article that was written at or below a sixth-grade level (2.1%). There were 7 websites below an eighth-grade reading level, which made up 14.8% of the total articles. Overall, readability was not significantly correlated with quality (rs = 0.14, P = .36).

There were 14 websites (29.7%) that were HONcode certified. Sites without HONcode certification had higher quality scores (8.33 ± 4.80) than sites with HONcode certification (6.18 ± 4.66), but this difference was not statistically significant (P = .15) (Table 3).

The average JAMA benchmark criteria score for all websites was 2.34 ± 1.11 of a maximum 4. JAMA criteria score was not significantly correlated with quality (rs = 0.02, P = .91).

Discussion

In this study, we found that the overall quality of information was incomplete, because websites had an average score of 6.47 of 20 total points. The Internet has become a part of daily life for Americans, and more patients are using it for their health information needs. Patients are even searching the Internet as their first source of health information, before physician consultation.39, 40 However, there is concern about the quality of health information that patients encounter. Many studies29, 34, 35, 41, 42, 43 have shown that the quality of online health information, particularly in orthopaedics, is substandard. This may negatively influence patients' expectations about their diagnosis and treatment plan and change the dynamic of the patient-physician relationship.16, 17 The effect of incomplete health information on those with rotator cuff disease is problematic because it is the most common disability of the shoulder. Despite this fact, the information that was provided by the assessed websites, although incomplete, was indeed helpful and valuable to patients seeking to learn about rotator cuff repair.

This study has shown that many websites failed to explain salient postoperative topics that could affect one's expectations of the requirements for a successful recovery. Only 13% offered advice for when to consult their surgeon after surgery, and 12% explained the effect of a patient's general health conditions (i.e., smoking, diabetes, etc) on recovery. In contrast, 60% of websites mentioned the signs and symptoms of a rotator cuff tear, and more than 40% of websites explained the surgical options for treatment. The lack of discussion on the breadth of information pertinent to a successful rotator cuff repair could cause patients to form unrealistic expectations for their recovery.

The analysis of reading level showed that only 1 article met a sixth-grade reading level (2.1%), which is similar to results published by other authors.23, 24, 28, 45 Zero websites met the recommended fifth-grade reading level for healthcare information recently proposed by the Joint Commission. These findings are especially concerning, given that Medicare insures the largest population of patients at risk for rotator cuff tears in the United States, and the average reading level for a Medicare beneficiary is at a fifth-grade level. Despite the failure of websites to meet the reading level requirement, we did not find reading level to be significantly correlated with quality. This is in contrast to prior studies23, 24, 27 and could mean that information that is more difficult to read does not simply equate to a better quality source of information.

The average JAMA score in this study was 2.34, which is relatively higher than scores from previous studies (1.43-2.1) in the orthopaedic literature.23, 30, 32, 45 However, a significant relationship between JAMA score and quality was not found, and thus website transparency and disclosure of sources does not necessarily enhance the quality of information.

The percentage of HONcode-certified websites was evaluated to assess the reliability and credibility of online information on rotator cuff tears and determine its relationship to quality. It was found that 29.7% of websites were HONcode certified, which is higher than investigations on Tommy John injuries (3.5%), femoroacetabular impingement (8%), total ankle replacement (8%), and anterior cruciate ligament reconstruction (17%). The higher number could be attributed to our publication being more recent and increasing awareness of the importance of certification. Interestingly, HONcode-certified websites had lower quality, although this statistic was not statistically significant. Furthermore, 5 of the bottom 6 websites were HONcode certified. These articles came from health information services such as “WebMD, Mayo Clinic, Healthline, and MedlinePlus,” all of which likely possess greater capabilities for search engine optimization and can generate additional “off-topic” articles for a given search. Although HONcode certification is an indicator of credibility, future authors researching the quality of online information should continue to study it to better define its relationship to quality.

Dalton et al performed an evaluation of websites on rotator cuff tears and evaluated quality of treatment with the DISCERN instrument. They found that the quality of available information is poor and the readability is inappropriately high. This study has similar findings using a novel scoring system to specifically query topics related to rotator cuff repair, which provides further confirmatory findings that online resources for patient education continue to be lacking.

Limitations

There are several limitations that must be mentioned in the context of the presented results. First, the scoring system used covers the breadth of information on rotator cuff repair, but it is not validated and standardized. Moreover, the study results are only applicable insofar as the search terms used, “rotator cuff repair” and “rotator cuff surgery,” and the date in which they were entered. The Internet is constantly changing, and it is possible new search results and ranks could be generated on another search date. Although this could be true, similar results published by Dalton et al. a few years before this publication makes this unlikely in the near future. Also, it is possible that websites outside the top 25 could have contained high-quality information. However, it has been shown that Internet consumers only use the first 10 search results, and therefore, the search was restricted to be consistent with other investigations of Internet information.23, 30, 31, 52, 53

Conclusion

The quality of patient-level information on rotator cuff repair on the Internet is both incomplete and written at a reading level higher than current recommendations. Information quality is not significantly correlated with reading level or JAMA criteria and does not depend on the search term used or HONcode certification.