Impingement on the internet: evaluating the quality and readability of online subacromial impingement information.

Patients increasingly access the internet to learn about their orthopaedic conditions. Despite this, online information may be unregulated, of questionable quality and difficulty to read.
OBJECTIVES: Therefore, this study aimed to evaluate the readability and quality of the online information concerning subacromial impingement syndrome.
METHODS: A search using Australia's three most popular online search engines was undertaken using the search terms 'subacromial impingement syndrome' and 'shoulder impingement'. The first 15 websites for each term were evaluated. Duplicates, advertisements and sponsored links were removed.The quality and readability of each website were calculated using the DISCERN and Flesch-Kincaid Reading Ease (FKRE) tools, respectively. The differences in quality and readability between each website type (healthcare/academic, commercial, news outlet, charitable/not-for-profit, layperson, government) was assessed using analysis of variance. The correlation between quality and readability was assessed using the Pearson correlation coefficient.
RESULTS: The majority of 35 unique websites analysed were of 'poor'/'fair' quality (determined via the DISCERN instrument) and 'difficult' readability (per the FKRE tool), with no correlation established between the scores. There was no statistically significant difference in quality across website types, however layperson, news outlet and government websites were found to be significantly more readable than alternate website categories (p<0.05).
CONCLUSIONS: We determined that much of the online information concerning subacromial impingement syndrome may be difficult to read and/or of poor quality. By recognising the shortcomings of information accessed by patients online, it is hoped clinicians may be prompted to better educate their patients. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Patients increasingly utilise the internet to learn more about their musculoskeletal conditions, such as subacromial impingement syndrome.

Despite this increase in usage, much of the information patients consult is unregulated.

This means much of the information patients consult may be inaccurate and difficult to read.

We sought to evaluate the ‘readability’ and ‘quality’ of online information concerning a common musculoskeletal condition, being subacromial impingement syndrome.

Much of the information consulted by patients on the topic of subacromial impingement syndrome are seen to be of poor quality and difficult to read.

It is hoped our paper will prompt higher quality and more comprehendible online resources on the topic, in order to aid patient understanding both inside/and outside of the clinic.

Introduction

Shoulder pain is a common and often debilitating complaint. For example, up to one-third of people in the community are reported to contend with shoulder pain at any given time, with prevalence observed to increase with age.1 2 Of this, subacromial impingement syndrome (SAIS) is believed to account for the most common cause.3 SAIS broadly refers to the painful entrapment of soft tissues, (often the subacromial bursa, supraspinatus muscle or portions of the bicipital tendon), between the humeral head and coracoacromial arch, that in particular occurs during elevation of the arm and potentially in the absence of preceding trauma.2 4 The model of soft-tissue impingement is a common hypothesis for chronic, mechanical shoulder pain, and is thought to be chiefly driven by the repetitive mechanical conflict of various structures of the shoulder joint.2 5

Despite popularisation of the term, alternate generators of degenerative shoulder pain have been proposed since the model was first popularised by Neer’s seminal 1972 paper.5 For example, some authors suggest the term ‘impingement’ is too broad to adequately account for the many pathophysiological entities contributing to degenerative shoulder disease such as rotator cuff pathology, and the resultant inflammatory and vascular changes.6 7 Consequently, SAIS today is seen as a ubiquitous but potentially disputed term referring to a broad spectrum of pathology, underpinned by a multifactorial pathogenesis.8 Owing to this evolving understanding, consensus regarding the topic can be difficult to decipher for clinician and patient alike.8

Therefore, it may not be surprising that many patients (such as those with SAIS) choose to access the internet for additional medical information to clarify their understanding of SAIS and other conditions. This trend has increased exponentially among Australian healthcare consumers in the last two decades.9 Likewise, increasing numbers of clinicians publish educational resources online to educate their patients.10 11 In orthopaedics in particular, the internet is fast becoming a popular repository of patient information.12

Despite more orthopaedic patients using the internet, the instructive value derived from online resources may be limited by poorly readable and low-quality materials.13 14 Prior authors have raised concerns regarding the readability and quality of online orthopaedic information.12 14–18 For example, more than 86% of online resources concerning general elective orthopaedics are written above recommended readability levels, with the majority being adjudged to be of poor educational quality.17

However, to our knowledge, no such study has evaluated online resources concerning SAIS. Consequently, the purposes of our study were to analyse the readability and quality of online SAIS information, to determine if there exists a correlation between the two, and to determine if they differ between website types. We hypothesised that readability and quality would not be correlated throughout websites assessed.

Methods

Search strategy

The use of search engines and search terms were designed to replicate the search strategy undertaken by a typical Australian patient. Search results were retrieved via the three most popular search engines in Australia: Google (www.google.com.au), Bing (https://www.bing.com/?cc=au) and Yahoo! (au.yahoo.com). These three search engines combined represent 98.2% of Australian online search volume.19 Likewise only the first 15 websites retrieved for each search term were assessed. This is because most internet users access websites encountered on the first page of a search engine.20

Our search strategy is summarised in figure 1. The terms selected were ‘subacromial impingement syndrome’ and the abbreviated, ‘shoulder impingement’. Searches were undertaken in December 2019. Excluded were advertisements, sponsored links, duplicate websites, websites without treatment information and gated content. Websites encountered were grouped into one of six categories (Government, News outlet, Layperson, Charitable/Not for Profit, Commercial and Healthcare/Academic). For this study, layperson websites were defined as resources such as patient testimonials and informal blogs with no obvious or declared commercial, governmental or professional affiliations. This yielded 35 unique websites which were evaluated for readability and quality.

Figure 1

Flow chart defining search strategy and exclusion criteria.

Assessment of information quality

Analysis of website quality was undertaken using the freely-accessible DISCERN instrument.21 The DISCERN instrument has been validated for use in healthcare and employed to assess a variety of clinical resources.22–24 The tool appraises materials across 16 domains, with all criteria being rated from 1 to 5, (except for question 2, which is scored from 0 to 5), with resulting scores summed.24 Resources are then categorised via previously published groupings, ranging from ‘poor’ (≤15) to ‘very good’ (≥80).24 25

To ensure the DISCERN tool was used in a reliable way, the same initial reviewer repeated quality assessments for a random assortment of 15 websites 1 week later. Intrarater reliability was then assessed via the intraclass correlation (ICC) coefficient. To gauge inter-rater reliability, another blinded reviewer was assigned to evaluate 15 websites at random. Scores were then categorised and compared with the first rater’s scores via Cohen’s quadratic rated kappa (κ). This method was employed in a seminal study that validated use of the DISCERN instrument.24

Assessment of information readability

The clarity or ‘readability’ of each website was evaluated via the Flesch-Kincaid Reading Ease (FKRE) score. This calculation considers the average frequency of syllables, words and sentences of a text as a surrogate of its complexity26 27:

Scores range from 0 to 100, with lower scores denoting more complicated language and higher scores reflecting easier readability. FKRE results were then sorted per previously described categorisations.28 To minimise the chances of human-error, an online FKRE calculator was used.29

Statistical methods

Data analysis was performed using Microsoft Excel, IBM SPSS Software (Build 1.0.0.1275), and SAS V.9.4. The extent of correlation between FKRE and DISCERN scores was evaluated via the Pearson correlation coefficient. The differences between website types in FKRE and DISCERN scores were analysed using analysis of variance (ANOVA) models. The overall association between website type and score were assessed using the F-test from the ANOVA model at the two-sided 5% significance level. If the overall F-test was statistically significant then the pairwise differences between the website types were assessed using two sided t-tests at the 5% significant level. The assumptions of the ANOVA model were assessed using residual plots and Levene’s test for homogeneity of variances.

Patient and public involvement

Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

Results

Readability

Readability as defined by FKRE scores varied from 73.3 to 14.4 (ie, from ‘easy’ to ‘difficult’ readability). FKRE mean was 45.62 (SD=14.41). Of the 35 websites included for analysis, only 1 qualified as having ‘easy’ readability (71–100), while 4 were adjudged to be of ‘standard’ readability (61–70) and the remaining 31 were ‘difficult’ to read (0–60). Both readability and quality scores can be visualised via table 1.

Table 1

Summary of DISCERN and FKRE scores

Instrument score	Websites
DISCERN score
Poor (15–28)	12
Fair (29–41)	12
Good (42-54)	9
Very good (55–67)	2
Excellent (68–80)	0
Total	35
FKRE score
Difficult (0–60)	30
Standard (61–70)	4
Easy (71–100)	1
Total	35

FKRE, Flesch-Kincaid Reading Ease.

There was a statistically significant difference in the FKRE scores between the website types (ANOVA, F=4.92, p=0.0022). Layperson websites were significantly more readable than commercial, charitable/not for profit and healthcare/academic websites (ANOVA, t-tests, p<0.05), while news outlet and government websites were significantly more readable than healthcare/academic websites (ANOVA, t-test, p<0.05).

Quality

DISCERN scores of the 35 websites analysed varied from 61 to 21. The mean score was 38.09 (SD=11.21)—which would correlate with ‘fair’ quality.25 30 According to accepted categorisations of DISCERN data, 12 websites were considered to be of ‘poor’ quality (15–28), 12 were ‘fair’ (29-41), 9 were ‘good’ (42-54) and only 2 were ‘very good’ (55–67). None of the websites evaluated achieved an ‘excellent’ score (68–80).

There was no statistically significant difference in DISCERN scores between website types (ANOVA, F=1.67, p=0.1734). The assumptions of the ANOVA models for both the quality and readability assessm

ents were found to be satisfied using residual plots and Levene’s test of homogeneity of variances. The residual plots and Levene’s test from the ANOVA model showed that the residuals were approximately normally distributed, and that there was no strong evidence that the variances for each website type were unequal (p>0.05).

Association between quality and readability

There was no significant correlation observed between the DISCERN and FKRE scores for the websites analysed. This was corroborated via the Pearson’s correlation coefficient calculated at −0.07260 (R2=0.0053) (p>0.05).

Reliability

The degree of intrarater and inter-rater reliability were acceptably high in our investigation. Intrarater reliability as evaluated by the ICC was calculated at 0.97 (95% CI 0.93 to 0.99), correlating with near perfect agreeance between DISCERN scores 1 week apart. Inter-rater reliability calculated via Cohen’s quadratic weighted kappa was calculated to be κ=0.65, (95% CI 0.44 to 0.85). This number exceeded the minimum acceptable degree of inter-rater agreement (κ=0.4) defined by the original creators of the DISCERN instrument24 and evidences appropriate usage of the tool by both raters in our study.

Discussion

Providing SAIS patients with high-quality and easily comprehendible educational resources is important. It is known that appropriately delivered patient education may reduce hospital readmission rates, promote patient adherence with proposed treatments and foster more satisfactory clinical outcomes.31 32 Conversely, information of lesser quality, or that which is unable to be easily interpreted by the patient may cloud decision-making and lead to unsatisfactory outcomes.33 This is perhaps particularly important for patients seeking to learn about SAIS, which is a commonly used, but increasingly disputed explanatory model of shoulder pathology.6 7 Patients must, therefore, be provided with information that accurately encapsulates clinical information, while ensuring it can be understood by its intended audience. As such, both quality and readability are important determinants of the value of an educational resource.

To this end, Australian patients are increasingly utilising the internet in search of useful information related to their health. This trend is supported in no small measure by the significant uptick in household internet access between the first two decades of the 21st century.9 For example, in 2017, nearly 90% of Australian households reported internet access, representing a 30% increase from 2009 census data.9 Beyond availability, other features of internet technology support this near ubiquity. Information obtained is often readily accessible, commonly available and free of charge.33 Indeed, popularisation of the term ‘Dr Google’ reflects the increasing trust patients invest in health information retrieved online. Despite this, much of the information disseminated online is unregulated. This makes it difficult to assess the degree of clarity, accuracy and educational value of online resources consulted by patients.34

We observed variability in both the quality and readability of online resources related to SAIS. Approximately two-thirds of websites evaluated (24 of 35 websites) were deemed to be of ‘poor’ or ‘fair’ quality via the DISCERN tool. Our results indicate that many the quality of many resources were limited by failing to elaborate on accepted treatment alternatives, a conditions natural history and neglecting to encourage shared decision making between patient and clinician.

Likewise, 86% (31 websites) of websites analysed were determined to be of ‘difficult’ readability using the FKRE score. This potentially draws into question the utility of information Australia patients consult on the topic of SAIS.

Furthermore, we observed that online SAIS information differs depending on where it is hosted. This is evidenced by a statistically significant difference in readability seen to exist between the website types. Post hoc analyses determined news outlets and government websites were significantly more readable than healthcare/academic websites, while layperson resources were significantly more readable than commercial, charitable and healthcare/academic websites.

Is there an association between readability and quality?

No compelling correlation was observed between the two measures (R2=0.0053). Thus, findings thus corroborate with those of similar quality and readability assessments employed in other domains of surgery.35 36

It is known that patients with musculoskeletal pathology are at increased risk of having reduced health literacy.37 This illustrates the importance of providing this patient population with accurate and digestible healthcare information. Despite this, our inquiry determined that the majority of websites related to SAIS are of ‘poor’/’fair’ quality and ‘difficult’ to read. This potentially constitutes a missed opportunity with respect to patient education and may complicate the task of the clinician, who must seek to redress any knowledge gaps in the clinic.16

What were our study limitations?

Our study has several limitations. First, only three search engines were utilised. While this was based on the current popularity of these websites in Australia, online website trends can be expected to vary across geography and time. Furthermore, while many patients might consult search engines for health-related information, this is obviously not true for all. Our study also failed to consider the myriad highly relevant websites that were not accessible via our strategy, yet nonetheless useful for patients. This includes websites requiring a login or paid access, including UpToDate and Medscape. Despite their non-inclusion, these websites are no less valid sources of patient information and might harbour concise and quality material.

Likewise, our study only considered the first 15 website results for each search term. This methodology overlooked high-quality resources that were not well indexed. As many search engines index results based on incoming links, newer websites might be overlooked, as they might not have established many connections. Despite this, our methodology was used as it is known that 75%–90% of internet users don’t consult beyond the first page of search engine results.20 Hence, it is hoped our analysis represents a reasonably accurate approximation of the patient experience.

Finally, there were also shortcomings with the tools used in our analysis. For instance, as the FKRE score considers syllables and the number of words in a sentence, information for conditions such as ‘SAIS’ might score poorly compared with conditions with shorter names. Likewise, the DISCERN tool is susceptible to rater bias and is also heavily weighted towards management information, with half of all criteria examining this aspect of a resource. This penalises otherwise useful resources that consider alternate aspects of a condition such as aetiology and natural history, rather than treatment.

Conclusion

SAIS is a debilitating spectrum of pathology afflicting many patients. Increasingly, many such patients turn to the internet to learn more about the condition and its management. However, much of online information relied on is found to be inaccurate in its description of the aetiology of shoulder pain and available treatments, as well as being of poor quality or difficult to read. Our investigation suggests that much of the online information related to SAIS is of ‘poor’/’fair’ quality and ‘difficult’ readability. As such, we suggest clinicians probe their patient’s understanding of SAIS in the clinic and address knowledge deficiencies accordingly. In addition, it is hoped that our study inspires clinicians to publish high-quality information that is easily comprehendible by patients.