Quality and readability assessment of websites related to recurrent respiratory papillomatosis.

OBJECTIVE: Recurrent respiratory papillomatosis (RRP) is a rare disease for which a limited number of information sources for patients exist. The role of the Internet in the patient-physician relationship is increasing. More and more patients search for online health information, which should be of good quality and easy readable. The study aim was to investigate the quality and readability of English online health information about RRP. STUDY
DESIGN: Quality and readability assessment of online information.
METHODS: Relevant information was collected using three different search engines and seven different search terms. Quality was assessed with the DISCERN instrument. The Flesch Reading Ease Score (FRES) and average grade level (AGL) were determined to measure readability of the English websites.
RESULTS: Fifty-one English websites were included. The mean DISCERN score of the websites is 28.1 ± 9.7 (poor quality); the mean FRES is 41.3 ± 14.9 (difficult to read); and the mean AGL is 12.6 ± 2.3.
CONCLUSION: The quality and readability of English websites about RRP is alarmingly poor. LEVEL OF EVIDENCE: NA. Laryngoscope, 127:2293-2297, 2017.

INTRODUCTION

Recurrent respiratory papillomatosis (RRP) is a rare illness caused by the human papilloma virus.1 The disease is characterized by multiple exophytic lesions of the mucosal squamous epithelium, called papillomata. The most common symptom is hoarseness, particularly if the lesions are located on the true vocal folds.1 The disease has an unpredictable and sometimes recalcitrant course. Treatment of RRP is symptomatic: there is no known curative therapy; it is based on the surgical removal of the papillomata.2

The Internet is a very common source of information worldwide. Of the entire world population, 46.4% the Internet.3 In Europe and North America, respectively 73.5% and 87.9% of the population uses the Internet.3 The Internet is also an important source of health information for patients.4, 5, 6 In 2014, 72% of the U.S. Internet users searched for health information online.7 It is estimated that 12.5 million health‐related searches worldwide are performed on the Internet every day.8 In 2002, only 13% of otorhinolaryngology patients used the Internet to obtain information before a consultation9; however, in 2011, 37% of the patients had accessed the Internet prior to their appointment.10

The impact of the Internet on the patient–physician relationship has been discussed extensively.11, 12, 13 The role of the patient has changed from a passive recipient of health information to an active consumer.12 The Internet can be used to strengthen the patient–physician relationship, and physicians should guide their patients to high‐quality websites.14 Because the Internet potentially influences patients' treatment choices; it is important that patients receive reliable health information online to reduce the risk of making incorrect clinical decisions based on Internet content.15 This endorses the importance of both quality and readability assessment of online health information.

The aim of this study is to investigate the quality and readability of online patient information about RRP.

MATERIALS AND METHODS

Google.com (Google Inc., Mountain View, CA, USA), Yahoo.com (Yahoo! Inc., Sunnyvale, CA, USA), and Bing.com (Microsoft Corp., Redmond, WA, USA) are the three search engines that were used. The three most used search engines worldwide are Google (71%), Baidu (13%) (Baidu Inc., Beijing, China), and Yahoo (7%).16 Because Baidu is exclusively available in Asian languages, the search engine was not applicable for this study. Hence, the fourth most used search engine was selected, which is Bing.com (market share of 7%).16

The authors established the search terms used to obtain the relevant websites on RRP. Consensus was reached through consultation. To collect English websites containing relevant patient information, the search terms “Recurrent Respiratory Papillomatosis”, “laryngeal papillomatosis”, “laryngeal papilloma”, “larynx papillomatosis”, “larynx papilloma” were used. In addition, the search terms “wart throat” and “wart vocal cords” were used to simulate patients' search behavior.

Searches were performed using Google Chrome v. 46.0 (Google Inc., Mountain view, CA, USA). The browser was set to incognito mode to prevent the search engine from showing personalized results. Location and language settings were set to default.

Inclusion and Exclusion

Each search term was entered into the three search engines. Only the first 20 hits were collected because it is natural behavior to not look at search results past the first page.17, 18 Sponsored links and advertisements were ignored.

Websites were excluded for the following: not containing information about RRP for patients; not written in English; requiring an account/payment to view the content; being a discussion forum; being a scientific article; being a PowerPoint presentation or video; and being dead links or security warnings.

Websites that were found using multiple search terms were marked as a duplicate. The calculation of the average scores is based on unique hits, but nonunique hits were used to compare search engines and terms. The term nonunique hits is here defined as “the number of hits before the removal of duplicates.”

The websites were divided into four different categories by the authors: governmental, commercial, nonprofit, and university/hospital. Websites stating that they were intended for layperson or patient readership were classified as for laypersons.

The authors have no relationship or conflicts of interest with any of the evaluated websites.

Quality Assessment

The DISCERN instrument was utilized to determine the quality of the selected websites. DISCERN is a reliable and valid tool for assessing the quality of written health care information.19 It has been used to assess quality of health information about many different diseases and treatments,20, 21, 22, 23, 24, 25 including various otorhinolaryngological illnesses and interventions.26, 27, 28, 29, 30, 31

The instrument consists of 16 questions divided into three sections (possible range 15–80) (Table 1). The first section, eight questions, addresses the reliability of the information and tests if the information could be trusted as a source of information about treatment options. The second section, seven questions, deals with specific information about the treatment options themselves. The third section, one question, is an overall quality rating.32 Scoring was performed by m.r.m.sg. and o.s.d.dg..

Table 1

DISCERN Score With Corresponding Quality Level.

DISCERN Score	Quality Level
< 27	Very poor quality
27 – < 39	Poor quality
39 – < 51	Fair quality
51 – < 62	Good quality
> 62	Excellent quality

Readability Assessment

Websites were assessed for readability using an online tool on www.readability‐score.com. This tool determines the Flesch‐Kincaid Reading Ease Score (FRES) and average grade level (AGL). The FRES is based on the number of words per sentence and the number of syllables per word, with FRES = 206.84 – (1.015 × average sentence length) – (84.6 × average number of syllables per word).33 Possible outcome is between 0 and 100, from very hard to understand to easy to understand. A FRES of 95 would indicate that a text is very easy to understand; a score of 65 suggests plain English; and a score of 15 indicates that a text is very difficult to comprehend.33 Table 2 displays the FRES in terms of difficulty and American school level.

Table 2

Flesch‐Kincaid Reading Ease Score With Corresponding Readability and Grade Level.

FRES	Difficulty	School Level (American)
90–100	Very easy	5th grade
80–90	Easy	6th grade
70–80	Fairly easy	7th grade
60–70	Plain English	8th/9th grade
50–60	Fairly difficult	10th–12th grade (high school)
30–50	Difficult	College
0–30	Very difficult	College graduate

FRES = Flesch‐Kincaid Reading Ease Score.

Apart from the FRES, the tool also generates the AGL. The AGL is the average of five different methods to determine the grade level. The methods are Flesch‐Kincaid Grade Level, Gunning‐Fog Score, Coleman‐Liau Index, SMOG‐index, and Automated Readability Index. All the above formulas produce an index that corresponds with the grade level in American education. The United States Department of Health and Human Services recommends that health information readability does not exceed sixth to seventh grade level.34, 35, 36, 37

Data Collection

Inclusion and exclusion of English websites was performed between September 18 and 24, 2015. The quality assessment was done on October 1 and 2, 2015. Readability assessment took place on October 22 and 23, 2015.

Statistical Analysis

Statistical analysis has been performed using IBM SPSS Statistics 22. The single measures Intraclass Correlation Coefficient was used to measure inter‐rater reliability (absolute agreement). The correlation between DISCERN and FRES, DISCERN score and AGL, and DISCERN and Douma score was determined using Pearson's correlation coefficient. A multiple linear regression (method: enter) was performed to determine whether there are predictors for a high DISCERN score. Dependent variables were language, FRES/Douma score, and website category. One‐way analysis of variance (ANOVA) was performed to compare the information found with the three search engines and the seven search terms. One‐way ANOVA was also performed to compare information of the four different categories. P < 0.05 was considered as statistically significant.

RESULTS

The systematic search yielded 420 English hits. Two hundred and sixteen websites were excluded, leaving 204 websites, of which 188 were duplicates. Ultimately, 51 unique websites were assessed (Fig. 1). Websites were divided into the different categories, as mentioned above (Table 3). Sixty‐nine percent (35 of 51) of websites were intended for lay readership.

Figure 1

Included and excluded websites and the reason why websites were excluded. [Color figure can be viewed in the online issue, which is available at www.laryngoscope.com.]

Table 3

Number of English Websites per Categorya and their Average DISCERN, Flesch‐Kincaid Reading Ease Score, and Average Grade Level.

Category	Amount (n)	DISCERN	FRES	AGL
Governmental	3	27 ± 10.0	50.1 ± 6.1	10.6 ± 0.5
Commercial	17	27.2 ± 12.4	45.7 ± 15.1	12.5 ± 1.9
Nonprofit	12	31.7 ± 9.3	35.5 ± 15.2	12.9 ± 2.3
University/hospital	19	26.8 ± 7.1	39.6 ± 14.5	13.0 ± 2.6

Highest scores are bolded.

Governmental, commercial, nonprofit, university/hospital.

AGL = average grade level; FRES = Flesch‐Kincaid Reading Ease Score.

Interrater Reliability of DISCERN Scoring

To determine exact inter‐rater agreement, the Intraclass Correlation Coefficient was measured. Single measures Intraclass Correlation Coefficient (absolute agreement) was calculated at 0.843 (P < 0.010), which indicates strong agreement.

The mean DISCERN score was 28.1 ± 9.7 (poor). With a DISCERN score of 55.5 (good), the website http://emedicine.medscape.com/article/865758‐overview scored highest.

The mean DISCERN score of nonprofit websites was highest of all four categories: 31.7 ± 9.3 (poor), as shown in Table 3. University or hospital websites scored lowest, with a mean DISCERN score of 26.8 ± 7.1 (very poor). Twenty‐six websites were of very poor quality; 17 websites were of poor quality; seven websites were of fair quality; and one website was of good quality. Not a single website scored high enough to be marked as excellent. All websites are shown in Supporting Appendix S1.

Readability assessment of the websites by the FRES and the AGL are shown in Table 3.

The mean FRES of the websites was 41.3 ± 14.9 (difficult). The mean AGL was 12.6 ± 2.3. The website easiest to read was https://www.urmc.rochester.edu/encyclopedia/content.aspx?contenttypeid=134&contentid=239, with a FRES of 69 (plain English) and an AGL of 8.4.

As shown in Table 3, of the English websites, the governmental websites were best readable, with a FRES of 50.1 ± 6.1 (fairly difficult) and an AGL of 10.6 ± 0.5. Nonprofit websites scored the worst on readability, with a FRES of 35.5 ± 15.2 (difficult) and an AGL of 12.9 ± 2.3. There was no significant difference between the four different categories in the mean DISCERN score (P = 0.554), FRES (P = 0.210), and AGL (P = 0.261).

Predictors of DISCERN and Readability Statistics

There was no significant correlation between the DISCERN and FRES score (r = −0.094 (P = 0.514)) and between the DISCERN score and AGL (r = 0.084 (P = 0.560)).

Predictors for DISCERN Score

Multiple linear regression was performed to determine whether there are predictors for a high DISCERN score. FRES, AGL, and website category did not predict for the DISCERN score (R2 = 0.049, F = 0.463 (P = 0.801)).

DISCUSSION

The quality and readability of online health information about RRP is substandard. Overall, the mean DISCERN score of the websites showed poor quality. The majority of English written websites, 26 out of 51, were qualified as very poor. Only one website was of good quality, and none scored high enough to be qualified as excellent. None of the websites met the recommended values of readability. The mean FRES indicates difficult readability and the mean grade level is 12.6th grade.

The use of the Internet by patients has increased dramatically over the past years.37 One would expect that this development continues today. It was shown that the patients already seek online information because they were inadequately informed about their disease; there was lack of time for explanation; the physician was unwilling to explain; the patient was ashamed to ask questions; or the physician did not succeed to provide comprehensible information.38 Therefore, the importance of good quality online health information has been emphasized by Clarke et al. by stating that ensuring the availability of valid, usable, and accessible information is a priority.15 To reduce the risk of patients making detrimental treatment choices based on online health information, it is important that patients receive reliable information online.

To the best of our knowledge, this is the first study to investigate the quality and readability of websites related to recurrent respiratory papillomatosis. It aims to elucidate the current situation in terms of availability and quality of online information about RRP and to create awareness among physicians. Moreover, this study could lead to collaboration of the different information centers to improve the quality and availability of the information.

Interestingly, of the 51 included websites, 26 were of very poor quality, 17 were of poor quality, seven were of fair quality, and only one website was of good quality. University or hospital websites had the lowest scores, whereas these websites often are supposed to be more reliable than commercial websites. In the case of RRP information, these websites were often disguised advertisements for certain treatments. Overall, websites were difficult to read (low FRES); to comprehend the websites, a fairly high school level (high AGL) was needed. The high density of poor quality websites is potentially dangerous for patients' knowledge on RRP.

No significant correlation was found between the DISCERN score and both FRES score and AGL. This means that a good‐quality website is not necessarily well readable and vice versa. This is especially problematic for semiliterate patients. High‐quality websites that are hard to read are no problem for highly educated patients, but semiliterate patients might have trouble understanding the information. For example, the website with the highest DISCERN score has a FRES of 25.6 and an AGL of 14.4. In other words, the English website with the highest quality rating is very difficult to read and requires a school grade level of 14.4 to be able to read the information easily. To be useful to all patients, readability of the assessed websites must be improved.

It is impossible to build a model to predict better DISCERN score‐based readability statistics and website categories. It follows that it is useless to guide patients exclusively to a certain type of website. All types of websites types of websites should improve—regardless of language, readability score, or category.

Because RRP is a rare disease, it is comprehensible that there is no abundance of high‐quality online information. However, the outcome of this study is alarming. The lack of good‐quality information should be an incentive for physicians to guide their patients in their search for reliable, intelligible, and correct information. Good‐quality websites should comply with the following general requirements: First, the aims of the website should be clear. A good website begins with explicitly stating what the website is about and whom it is meant for. In that way, patients instantly will know if the website contains the information they are looking for. Of the included websites together, only two websites had clear aims. Clearly, this is an aspect of most websites that needs improvement. Secondly, websites should be evidence‐based, which means that the sources (and their publication dates) that were used to compile the website should be clear. Thirdly, the website should provide additional sources of information and should refer to areas of uncertainty. Information on treatment options should accurately describe each treatment, their benefits and risks, and their impact on the patients' daily life.

CONCLUSION

Limitations

The DISCERN instrument, although being a reliable and valid tool to assess health information, has its limitations. The most important shortcoming of the DISCERN tool is that it does not take into account how the information is presented or how easy it is to navigate and find the information on a particular website. Furthermore, the DISCERN scoring has been performed by two researchers. It is not entirely clear if laypersons would assess the websites in the same way. However inter‐rater agreement between these two researchers showed strong agreement, confirming the reproducibility of this score.

Readability statistics, such as the FRES and grade levels, have been criticized. Some argue that readability statistics based merely on word and sentence length do not adequately reflect the complexity and readability of a text, but that this depends on more factors than just word and sentence length.39

Recommendations

Of all 51 websites evaluated, one (http://emedicine.medscape.com/article/865758‐overview) had good quality. None of the websites met the study's criteria for readability.

Webmasters of websites containing health information are recommended to adjust their websites according to the above‐mentioned criteria for good websites. Otorhinolaryngologists worldwide should consider the possibility of jointly making a website containing high‐quality intelligible information, in various languages, for patients and their partners.

Additional supporting information may be found in the online version of this article.

Supporting Appendix S1. Included websites with information on Recurrent Respiratory Papillomatosis with their LINK, mean DISCERN score, the three Discern sections, Flesch Reading Ease Score and the average Grade level.

Click here for additional data file.