The quality of health information on allergic rhinitis, rhinitis, and sinusitis available on the internet.

PURPOSE: The internet has become one of the most important media outlets used to obtain health information. Therefore, the quality of health information available on the internet is very important. We evaluated the quality of internet-derived health information on allergic rhinitis, rhinitis and sinusitis and compared these results to those of previous studies performed five years ago.
METHODS: The terms "allergic rhinitis (AR)", "rhinitis" and "sinusitis" were searched among the four most commonly used search engines in South Korea. These websites were evaluated according to the author, the Journal of the American Medical Association (JAMA) benchmarks, the DISCERN questionnaire and the Allergic rhinitis and its Impact on Asthma (ARIA) 2008 Update.
RESULTS: A total of 120 websites were obtained and analyzed. For all diseases, "Oriental physician" had the largest portion (almost half of all websites), followed by "Western physician". Based on analyses using the JAMA benchmark, "Attribution" and "Disclosure" were ignored in almost all surveyed websites. According to the scores of the DISCERN question, the majority of websites did not supply appropriate references for their health information, and information on the negative aspects of treatment such as risks and uncertainty was not provided in several websites. In an analysis based on the ARIA 2008 Update concepts, 65% of websites pertaining to health information on AR contained unreliable information.
CONCLUSIONS: The quality of health information on the internet was not acceptable. Thus, governmental regulation or control to improve the quality of health information is required.

INTRODUCTION

The internet has become one of the most important media outlets for obtaining health information.1 However, the quality of health information available on the internet is unclear. If incorrect information is spread via the internet, it can promote false beliefs. Concerns regarding the quality of health information on the internet have been increasing.2 Therefore, several studies evaluating the quality of health information on the internet have been performed globally.3,4,5,6,7,8,9,10,11,12,13,14,15 South Korea has one of the most developed internet systems. In 2012, the rate of use of the internet in South Korea reached 78%, which is greater than the global average of 36%. In 2013, South Korea ranked first among OECD nations in terms of the availability of wireless high-speed internet services.16 Therefore, evaluation of the quality of health information on the internet in South Korea could be used as a model for many developing countries and as a blueprint for controlling the quality of this health information. Five years ago, we evaluated the quality of health information obtained from 40 websites containing information on allergic rhinitis (AR).5 In this report, we evaluated the quality of health information from websites containing information on AR, rhinitis and sinusitis, and we compared these results to those obtained 5 years ago.

MATERIALS AND METHODS

This study applied a similar method as used in our previous report5; however, the number of search terms was increased from one to three. "AR", "rhinitis" and "sinusitis (rhinosinusitis)" were searched in the Korean language via the four most commonly used search engines in South Korea: Naver (www.naver.com), Daum (www.daum.net), Google Korea (www.google.co.kr) and Nate (www.nate.com). To ensure that the search conditions were similar to those used by the general population, we did not modify the search conditions. We used the top 10 websites per search term from each search engine. Overlapping websites and websites that did not contain health information on the search term were excluded. Based on these criteria, we obtained a total of 120 websites. These websites were evaluated between January and March of 2014. Evaluation was conducted by the first author alone. The authorship was categorized into five groups: 1) a Western physician, a registered medical practitioner certified by a Western medical university; 2) an Oriental physician, a practitioner of traditional Oriental medicine certified by an Oriental medical university; 3) a blogger, a person who maintained and updated a blog and was not recognized as a Western physician or Oriental physician; 4) a journalist, a writer for internet newspapers; and 5) others, individuals or organizations not belonging to any of the previous categories. Initially, we intended to include the "commercial agencies" category, as done in previous reports. However, no website included in this study was generated by a commercial agency.

Basic information on the websites was evaluated using the Journal of the American Medical Association (JAMA) benchmarks17 and the DISCERN questionnaire.18 JAMA benchmarks are comprised of four concepts: authorship, attribution, disclosure and currency. The concept of authorship required that the authors, contributors, credentials and affiliations are cited in the websites. The concept of attribution concerns the references, sources and copyright for the information. The concept of disclosure required that website ownership, sponsorship, advertising and conflicts of interest are disclosed. The concept of currency concerns the dates the information was updated on the websites.17 The DISCERN questionnaire is comprised of 16 questions assessing several aspects of the health information (Table 1). The score of each question ranged from 1 (the criterion was not achieved) to 5 (the criterion was achieved).18 The quality of health information was evaluated based on the scores obtained for the 16 questions. The DISCERN questions related to specific aspects of health information were assigned to four categories; the reference for the health information, negative aspects of treatment, neutrality of the author and effect of treatment (Table 2). The mean scores of the DISCERN questions from each category were calculated by the author.

Table 1

The 16 questions of the DISCERN instrument

DISCERN questions
1.	Are the aims clear?
2.	Does it achieve its aims?
3.	Is it relevant?
4.	Is it clear what sources of information were used to compile the publication (other than the author or producer)?
5.	Is it clear when the information used or reported in the publication was produced?
6.	Is it balanced and unbiased?
7.	Does it provide details of additional sources of support and information?
8.	Does it refer to areas of uncertainty?
9.	Does it describe how each treatment works?
10.	Does it describe the benefits of each treatment?
11.	Does it describe the risks of each treatment?
12.	Does it describe what would happen if no treatment is used?
13.	Does it describe how the treatment choices affect the overall quality of life?
14.	Is it clear that there may be more than one possible treatment choice?
15.	Does it provide support for shared decision-making?
16.	Based on the answers to all of the above questions, rate the overall quality of the publication as a source of information about treatment choices.

Table 2

Classification of the DISCERN questions

Category of the DISCERN question	DISCERN question
The references of the health information	4.	Is it clear what sources of information were used to compile the publication (other than the author or producer)?
	5.	Is it clear when the information used or reported in the publication was produced?
The negative aspects of treatment	8.	Does it refer to areas of uncertainty?
	11.	Does it describe the risks of each treatment?
	13.	Does it describe how the treatment choices affect the overall quality of life?
The neutrality of the author	6.	Is it balanced and unbiased?
	14.	Is it clear that there may be more than one possible treatment choice?
The effect of treatment	9.	Does it describe how each treatment works?
	10.	Does it describe the benefits of each treatment?
	12.	Does it describe what would happen if no treatment is used?

The Allergic rhinitis and its Impact on Asthma (ARIA) 2008 Update19 was used to evaluate the value of the AR-related information. The contents of those websites obtained using the search terms "rhinitis" or "sinusitis" were not evaluated in accordance with the ARIA 2008 Update. Evaluation was performed using the following eight categories: definition, symptoms, subdivision, severity, risk factors, mechanisms, diagnosis and management. We evaluated eight categories by citing the ARIA 2008 Update concept. Any concept not included in the ARIA 2008 Update was deemed unreliable information. The unreliable information was evaluated using the "Justification" code of the Health On the Net (HON) principles. The HON principle is one of the most frequently used codes for the ethics of health information on the internet. The information provided appropriate evidence, and obvious references were considered as fulfillment of the "Justification" code.20 Statistical analyses were conducted using SPSS ver. 12.0 (SPSS, Chicago, IL, USA). P value <0.05 was considered statistically significant.

RESULTS

A total of 40 websites were obtained using each search term, including "AR", "rhinitis" and "sinusitis", respectively. As a result, 120 websites were obtained and assigned to five groups according to the author. The website authorship proportions were evaluated for each disease. For all diseases, "Oriental physician" was the most frequent type of author of the websites found, accounting for almost half of all surveyed websites, followed by "Western physician" (Fig. 1). Other websites were often from posts from government organizations and some portal sites, for which the author was not identified and for which the websites were created by both Western and Oriental physicians.

Fig. 1

Study proportions according to authorship.

The citation frequency of the JAMA benchmark concepts for the websites was obtained. "Authorship" was cited on all websites created by Western physicians, Oriental physicians or journalists, while it was not cited on many of the websites created by bloggers or others. "Attribution" and "Currency" were cited more frequently on the websites created by journalists or bloggers than on those created by Western physicians or Oriental physicians. "Disclosure" was cited rarely on the surveyed websites (Fig. 2).

Fig. 2

Citation frequency using the Journal of the American Medical Association (JAMA) benchmark concepts according to authorship.

The mean score for all of the DISCERN questions was 1.98 (±1.28) out of 5, when the websites were evaluated as a whole. The websites were grouped by authorship type, and the mean scores of the DISCERN questions were calculated according to authorship. The results for the others websites were excluded from the statistical analyses based on authorship, because these consisted of websites from heterogeneous author types. The mean scores for all DISCERN questions according to authorship were 2.74 (±1.54), 1.69 (±0.98), 1.57 (±0.97), 1.68 (±1.11) and 2.47 (±1.65) for websites by Western physicians, Oriental physicians, bloggers, journalists and others, respectively. The score for Western physician websites was significantly higher than that for Oriental physician, blogger and journalist websites (P<0.05). The mean scores for the DISCERN questions were calculated according to the four categories. With regard to "the reference of health information" category, the mean scores for the four authorship type websites ranged from 1.28 to 1.46. There was no significant difference among the groups. Concerning "the negative aspects of treatment" category, the mean scores for Western physician, Oriental physician, blogger and journalist websites were 1.85 (±1.44), 1.00 (±0.00), 1.06 (±0.32) and 1.14 (±0.49), respectively. The score for Western physician websites was significantly higher than those for Oriental physician and blogger websites (P<0.05). With regard to both "the neutrality of the author" and "the effect of treatment" categories, the mean score for Western physician websites (3.94±1.16, 3.06±1.55) was significantly higher than those for Oriental physician (1.39±0.54, 1.98±1.00), bloggers (1.44±0.62, 1.75±1.02) and journalist (1.42±1.14, 1.81±1.17) websites (P<0.05) (Fig. 3).

Fig. 3

The mean score for the DISCERN questions according to the four categories. The asterisk indicates a statistically significant difference between two groups (P<0.05).

The mean cited numbers of ARIA 2008 Update concepts in all surveyed websites was 2.9 out of 8, with a median of 2 (range, 0 to 7 concepts). When the websites were grouped by authorship, the mean cited numbers were 4.5 for Western physician websites, 2.6 for Oriental physician websites, 2.2 for blogger websites, 3.0 for journalist websites and 2.5 for other websites. With regard to Western and Oriental physician websites, the citation frequencies for "definition" and "symptom" were more than 70% in the 2 groups. The citation frequencies for "mechanism", "diagnosis" and "management" decreased to 50%, 67%, and 67%, respectively, among Western physician websites. However, they decreased significantly to 6%, 6%, and 0%, respectively, among Oriental physician websites. Blogger and journalist websites also showed high citation frequencies for "definition" and "symptom", but showed low citation frequencies for "mechanism", "diagnosis", and "management" (Table 3). All Oriental physician websites included unreliable information concerning the "mechanism" or "management" of AR. Approximately half of the websites created by bloggers, journalists and others also included unreliable information concerning the "mechanism" or "management" of AR. Western physician websites included no unreliable information. Only one (6%) of the Oriental physician websites that contained unreliable information provided clear references for the unreliable information. The remaining websites did not supply evidence or references for the unreliable information and did not fulfill the HON code "Justification" criteria (Table 4).

Table 3

Citation frequency of the Allergic Rhinitis and its Impact on Asthma (ARIA) 2008 update concepts by authorship type

ARIA 2008 update	No. of websites (%)
	Western physician (n=6)	Oriental physician (n=18)	Blogger (n=5)	Journalist (n=7)	Others (n=4)
Definition	6 (100)	13 (72)	3 (60)	5 (71)	2 (50)
Symptoms	6 (100)	17 (94)	3 (60)	7 (100)	3 (75)
Subdivision	1 (17)	6 (33)	0 (0)	3 (43)	1 (25)
Severity	0 (0)	0 (0)	0 (0)	0 (0)	0 (0)
Risk factors	3 (50)	8 (44)	2 (40)	3 (43)	2 (50)
Mechanisms	3 (50)	1 (6)	1 (20)	1 (14)	0 (0)
Diagnosis	4 (67)	1 (6)	1 (20)	1 (14)	2 (50)
Management	4 (67)	0 (0)	1 (20)	1 (14)	0 (0)
Mean cited numbers of concepts	4.5±1.8	2.6±1.3	2.2±2.3	3.0±1.6	2.5±1.9

No, number.

Table 4

Unreliable information by the author

	No. of websites (%)
	Western physician (n=6)	Oriental physician (n=18)	Blogger (n=5)	Journalist (n=7)	Others (n=4)
No. of sites containing unreliable information	0 (0)	18 (100)	2 (40)	4 (57)	2 (50)
Details of the unreliable information
Mechanism	0 (0)	16 (89)	0 (0)	1 (14)	1 (25)
Management	0 (0)	18 (100)	2 (40)	4 (57)	2 (50)
No. of sites containing unreliable information yet satisfying the criteria of justifiability	-	1 (6)	0 (0)	0 (0)	0 (0)

No, number.

DISCUSSION

In this study, we evaluated the quality of health information provided to the general population using two methods that differed from those of other studies. First, we did not apply exclusion criteria in our search for websites. Searched websites were not excluded from analysis, except for those that were duplicated or did not contain health information on the search term. Second, to include the popular websites accessed most frequently by laypeople, only 10 websites per search term were obtained from a single search engine. If numerous websites for a search term were obtained from a single search engine, unpopular websites accessed rarely by laypeople would also be included in the analysis.

Authorship of the websites

We analyzed websites according to factors that can be recognized easily by laypeople. Therefore, we adopted authorship as an important criterion, which is an easily recognized factor and one of the most important for determining the quality of health information. The authorship of the surveyed websites was determined, and the quality of health information was evaluated according to authorship. Oriental physician websites comprised the largest proportion of all surveyed websites, accounting for approximately 50% of all websites. Western physician websites comprised the second largest proportion, approximately 20%. Since surgery was considered more frequently on sinusitis websites than on AR or rhinitis websites, the proportion of Western physician websites was increased slightly; however, Oriental physician websites accounted for the largest proportion. For the study evaluating AR conducted 5 years ago, the proportions of Western and Oriental physician authorship websites were 50% and 35% of the surveyed websites, respectively.5 In this current study, the proportion of Oriental physician websites increased to 45% of the total websites on AR, while that of Western physician websites decreased to 15%. The proportion of Oriental physician websites increased significantly over 5 years, and Oriental physician websites are the primary health information source on the internet.

There are approximately 100,000 Western physicians and 20,000 Oriental physicians in South Korea.21 Although Oriental medicine is recognized as an official form of medicine in South Korea, Western medicine comprises 5-fold as many physicians, and Oriental medicine accounts for a minor portion of the health care system in South Korea. Considering this fact, it was an unexpected finding that Oriental physician websites comprised a 2.3-fold larger proportion of the surveyed websites than that of Western physician websites. Although Oriental medicine represents a minor portion of the health care system, it has become a primary source of health information on the internet in South Korea, at least for otorhinolaryngology. This result may be attributed to the internet's ease of accessibility and high usage rate in South Korea.

In the United States, Oriental medicine is considered an alternative medicine. The effects that alternative medicine has had on public health in the United States cannot be ignored. According to national surveys, 34%-42% of Americans have used alternative medicine modalities in the past year.22,23,24,25 This tendency may not differ from that in other Western countries. Considering the effects of alternative medicine, it is possible that alternative medicine represents a considerable source of health information available on the internet in Western countries as the internet usage rate continues to grow, as has occurred in South Korea. At this time, no study has examined health information according to authorship in Western countries, even though many have evaluated the health information accessible on the internet. As the usage rate of the internet increases, studies that classify health information according to authorship and evaluate health information derived from alternative medicine sources may be needed in Western countries.

The DISCERN instrument and the JAMA benchmark concept

The DISCERN instrument has been used by several studies to evaluate the quality of health information on the internet. Evaluation using the DISCERN instrument typically evaluates the scores of all 16 DISCERN questions or classifies websites into five grades according to the mean score from all DISCERN questions.3,5,6,7,8,11 Determining the scores for all DISCERN questions may disperse the focus of analysis, and rating of the websites according to the mean DISCERN question score may result in loss of diverse information from the DISCERN instrument. Therefore, in this study, in addition to evaluating health information using the mean DISCERN question score, the DISCERN questions were assigned to four categories, and each category was evaluated according to the mean DISCERN score.

The mean score for all DISCERN questions was calculated as 1.98 out of 5, which is not an acceptable result. A previous study in the United Kingdom evaluated 124 websites on six ear, nose and throat diseases (cholesteatoma, sinusitis, tonsillitis, acute otitis media, epistaxis, and quinsy) using the DISCERN instrument. In that study, the mean DISCERN score was 2.44, which was better than our result.7 That study used only Google as a search engine and 6 search terms. Therefore, direct comparison between those results and ours is not possible. However, it remains possible that English websites or websites searched by Google may be of better quality.

We evaluated the quality of health information according to four categories. First, the citations of the referenced information was evaluated based on DISCERN questions 4 and 5. The mean scores for websites by four authorship types were each less than 1.5. This result was consistent with the results of the JAMA benchmarks concept "Attribution". The majority of websites (by any author) did not provide the proper references for their health information. Second, information on the negative aspects of treatment, such as risks, uncertainty and effects on quality of life, was evaluated using DISCERN questions 8, 11, and 13. The score for Western physician websites was the highest (1.85), which was also a disappointing score. Particularly, among Oriental physician websites, the scores for the negative aspects of treatment were 1.0 (criterion not fulfilled by the publication) in all surveyed websites. In other words, Oriental physician websites do not discuss the negative aspects of treatment. Considering that Oriental physician websites accounted for almost half of all surveyed websites, this is a major issue that requires immediate remediation. Third, the neutrality of the author was evaluated based on DISCERN questions 6 and 14. The score of Western physician websites in this category was approximately 4, which was higher than those in the other categories. However, the scores for the other 3 groups was less than 2. It was common in Oriental physician websites that the treatment performed by the author was introduced as the trade name (not the general name), and information on the treatment not performed by the author was not provided. In addition, considering that Oriental physician websites did not discuss the negative aspects of treatment, they recommended their own treatments without providing other treatment options. Lastly, information on the effect and efficacy of treatment was evaluated based on DISCERN questions 9, 10, and 12. The score for Western physician websites was 3.06, and those for other authorship type websites were less than 2.0. Overall, these results were not satisfying, despite being better than those for the other categories.

The ARIA 2008 update

Basic quality of information was evaluated using the JAMA benchmark concepts and the DISCERN questionnaire. Since there are international guidelines for AR, the ARIA 2008 update,19 which was proposed at the World Health Organization (WHO) workshop, was used to analyze the contents of AR-related websites. All Western physician websites addressed the "Definition" and "Symptom" concepts of the ARIA 2008 update. Other ARIA 2008 update concepts (excluding "Severity") were addressed in 50%-67% of Western physician websites. No Western physician websites contained unreliable information on any of these concepts. More than 70% of Oriental physician websites referred to the "Definition" and "Symptom" concepts of the ARIA 2008 update. However, "Mechanism" and "Diagnosis" concepts were found on only one Oriental physician website (6%), and "Management" was not found on any Oriental physician website. Unreliable information on "Mechanism" and "Management" concepts was found on 89% and 100% of Oriental physician websites, respectively. In other words, Oriental physician websites had the same concepts with western medicine about "Definition" and "Symptom" for AR, but they had the different concepts about "Mechanism" and "Management". There was no consistency among the "Definition", "Symptom", "Mechanism" and "Management" concepts for AR. Concerning the mechanism of AR, the concepts among Oriental physician websites differed. For example, one Oriental physician website proposed the dysfunction of lung, spleen and kidney as a mechanism of AR and another proposed an increase of toxins from the large intestine as a mechanism. The other proposed the consumption of "Ki (energy)" as a mechanism. These results agreed with a previous study performed 5 years ago.5 Concerning management, Oriental physician websites use the trade name and not the general name for a specific treatment. Therefore, it is unclear whether consistent treatments were used, and the treatments used by each Oriental physician could not be compared. Although the ARIA 2008 Update concepts used in this study were proposed at the WHO workshop, they were based on Western medicine. Since Oriental medicine uses different concepts regarding Western medicine diseases, the present study has limitations in its evaluation of the contents of Oriental physician websites using the ARIA 2008 Update concepts.

However, on Oriental physician websites, some aspects of disease were explained using Western medicine concepts, and other aspects were explained using Oriental medicine concepts. This may be attributed to the fact that explaining a disease solely using Oriental medicine concepts would not be sufficient for public understanding. Since the majority of Oriental physician websites showed no consistency among concepts to explain a disease, it is important to develop methods to explain the disease clearly using Oriental medicine concepts.

Among blogger and journalist websites, more than 60% contained "Definition" and "Symptom" concepts from the ARIA 2008 update, and approximately 50% contained unreliable information on the "Management" of AR. The majority of information on blogger and journalist websites was derived from the quoted opinions of Western or Oriental physicians, which may have contributed to this result.

Proposal for developing international criteria

Several studies have examined the quality of health information from the internet. Several criteria such as JAMA benchmarks, Information Score, Flesch-Kincaid Grade Level Readability Score, Quality Evaluation Instrument, Global Quality Score, and DISCERN were applied. Although the criteria used by many studies have differed from each other, the results have been similar. The quality of health information from the internet is not acceptable,3,4,5,6,7,8,9,10,11,12,13,14,15 nor were the results of this study. Therefore, an international effort to improve the quality of health information on the internet is required. First, international criteria and protocols to evaluate the health information available on the internet should be created. Since the criteria used differed among several studies, it was difficult to compare the results among studies and to propose a standard for health information. If international criteria and protocols are developed, the quality of health information in each country could be evaluated via the same methods to improve its quality. Second, websites containing proper health information should be verified and scored using these international criteria. This will help laypeople decide whether health information on the websites is reliable, which can improve the patient's right to self-decision.

CONCLUSIONS

The internet is a medium via which anyone can publish and access information easily. At this time, when health information is developing rapidly, no media can transmit vast health information more efficiently than can the internet. Therefore, if proper health information is transmitted to laypeople through the internet, it could be used to make treatment choices. However, this could be problematic if unverified health information is provided. Therefore, websites containing health information on various diseases must be surveyed, controlled and regulated. Efforts to improve the quality of health information on the internet could provide proper and appropriate information to consumers.