The quality of non-alcoholic fatty liver disease information resources for patients on the internet in Korea.

BACKGROUND/AIMS: Many patients uses the internet to obtain information about their diseases. However, there is increasing concern regarding the quality of internet information. Thus, we aimed to systematically evaluate the quality of websites containing educational information about non-alcoholic fatty liver disease (NAFLD) in Korea.
METHODS: Naver, Daum, and Google search engines were searched using the term "non-alcoholic fatty liver disease" in Korean. Two reviewers independently evaluated website quality using the quality evaluation instrument (QEI), which awarded websites scores for specific information on various aspects of NAFLD, as well as a five-point Likert scale (1-5), the DISCERN instrument, and a global quality scale (GQS).
RESULTS: Forty-seven websites met the inclusion criteria. We found that the quality of the internet information about NAFLD is generally poor. The mean QEI score with standard deviation was 10.31 ± 5.09 (range, 4 to 22), with only 17% of websites scoring higher than 10 points. The median GQS of the websites was 2.0, with no website achieving a score of 4 or 5. The QEI score was highly associated with the GQS score (r = 0.74, p < 0.01). For each DISCERN question from question 1 to question 15, the mean score was less than 3.
CONCLUSION: Overall, the internet health information for patients regarding NAFLD is poor and in need of much improvement. There is a need for institutional support, qualitative regulation of internet information, and development of an accreditation system to provide patients with internet health information of appropriate quality.

INTRODUCTION

Non-alcoholic fatty liver disease (NAFLD) is a broad term encompassing a spectrum of diseases, from non-alcoholic fatty liver (NAFL) to non-alcoholic steatohepatitis (NASH), and NASH cirrhosis. In the past, viral hepatitis and alcoholic liver disease were the main causes of chronic liver disease, but currently NAFLD is a leading cause of liver-related morbidity all over the world and the prevalence of NAFLD is increasing in Korea [1,2]. The prevalence and incidence of NAFLD are varied and based on the study subjects, diagnostic standards, and definition of NAFLD [1,3-6]. The global prevalence of NAFLD is approximately 25%. The highest prevalence was reported in the Middle East (32%), followed by South America (31%), Asia (27%), North America (24%), Europe (23%), and Africa (14%) [7].

Recently, the metabolic diseases associated with obesity have been rapidly increasing in Korea. NAFLD has a high prevalence (16% to 33%) and incidence rate (26 cases per 1,000 persons) [4,8]. Therefore, it is expected that advanced liver diseases, such as liver cirrhosis (LC) and hepatocellular carcinoma (HCC) will increase due to NAFLD in the future [9].

As with other chronic diseases, patient education by providing appropriate and sufficient information regarding NAFLD is valuable for self-care, compliance, formation of the patient-doctor relationship, the disease natural history and treatment outcome, and reducing personal and national medical costs. NAFLD is a disease that can progress to LC and HCC, and the effectiveness of pharmacotherapy in managing NALFD is still uncertain. Lifestyle changes, such as diet and weight control, are important in managing NALFD, which means that patient awareness of their disease severity as well as general knowledge regarding NAFLD are important for improving treatment compliance and the occurrence of regular follow-ups [1,10]. However, one study identified a very low awareness of NAFLD (19%) among patients at metabolic risk for the disease. In addition, if patients were told that they were at risk for NAFLD, a large proportion (73%) of the patients wanted to be educated about NAFLD, as well as access to written materials, contact with health care provider, and online learning tools as educational resources [11].

The internet is fast, accessible, and full of rich and diverse information content, with no significant limitations in terms of time and cost. In particular, Korea has a high internet penetration rate (greater than 90%), and people use the internet as a source for various information [12]. Approximately 85% of Korean internet users have used the internet to search for health information [13].

Several studies have demonstrated that the majority of patients with chronic disease use the internet to find health-related information about their medical conditions [14-16]. However, biased, inaccurate, and non-evidence based or not up-to-date information can result in adverse effects with regards to disease outcome, the patient-doctor relationship, and medical costs. In fact, according to new evidence, there is increasing concern regarding the quality of internet information that is inaccurate, inappropriate, or not up-to-date [17-19].

Currently, there are no studies evaluating the quality of internet information for patients with NAFLD. Thus, the aim of our study was to assess the quality of internet sources that contain educational information on NAFLD in Korea.

METHODS

In the evaluation of the quality of medical information on fatty liver disease websites, the quality of medical information was evaluated only for NAFLD except alcoholic fatty liver disease. Because alcoholic fatty liver disease is relate to alcoholism and abuse, it was judged that the insight and treatment compliance is lower than that of NAFLD patients and that interest and behavior are lower to obtain specific information about the disease.

Before the implementation of the study, a protocol was selected and developed that achieved the specific objectives of the study to evaluate NAFLD websites for: (1) the quality of the general information provided about NAFLD, using a quality evaluation instrument (QEI); (2) the quality of written information about treatment choices using the DISCERN instrument; and (3) the global quality scale (GQS), which assessed overall quality, flow of the website, ease of use, and usefulness to patients. The QEI and GQS were developed by referring to a study that conducted a quality review of internet information for other diseases [20,21]. Website information quality was evaluated independently by two reviewers (I.T.S. and H.I.J.), gastroenterologists with 2 years of experience, to ensure accuracy. Any differences or conflicts between two reviewers were mediated by B.K.J., a hepatologist who has been treating NAFLD patients for more than 10 years and has a special interest in NAFLD, to generate a single result for the website in question.

Internet search strategies

In our study, we identified websites created for patients and the public on NAFLD by searching with three search engines (Naver, Daum, and Google) using “non-alcoholic fatty liver disease” in Korean. Although Google is the most used search engine globally, the most used search engine in Korea in 2017 is Naver, followed by Daum and Google [22,23]. From April 7 to 13, 2018, a search was conducted using “non-alcoholic fatty liver disease” in Korean as the keyword using the Naver, Daum, and Google search engines, limiting the results to the first three pages for each search string and category. Finally, the website searched on April 16, 2018 was selected and evaluated. We screened the three pages of website results to review their relevance because recent studies have shown that 92% of people do not click results after the first page of search results, 62% of search engine users click on a search result within the first page of results, and 90% of search engine users click on a result within the first three pages of search results [22,24].

Naver and Daum search results are divided into several categories, unlike Google, which provides a sequential list of search results. As such, the Naver search engine provides three pages each for the encyclopedia, blog, website, post, and cafe categories. Additionally, the Daum search engine provides three pages each for the encyclopedia, blog, website, and cafe categories. The term cafe is a unique name for a search engine that refers to a website where people who are interested in a particular topic communicate information to each other.

Inclusion and exclusion criteria

The inclusion criteria to be eligible for review included: (1) websites covering health information and education about NAFLD and (2) websites focusing on patient education written in Korean. The exclusion criteria included: (1) websites that functioned primarily as portal websites leading to another site and or advertisement sites; (2) websites focused primarily on non-NAFLD illnesses, e.g., irritable bowel syndrome; (3) websites consisting of newspaper articles; (4) websites with content not intended for patients (e.g., academic databases and journal articles); and (5) YouTube videos. Most of the YouTube related to NAFLD were related to newspaper articles and advertisements, which are exclusion criteria in the data collection method.

Identified websites were grouped into five categories: (1) institutional (e.g., government, hospital, or university); (2) non-pharmaceutical commercial (e.g., sponsored sites or private medical sites); (3) pharmaceutical; (4) charitable (e.g., a nonprofit organization); (5) support (e.g., personal webpages or patient support groups); and (6) alternative medicine (e.g., nonorthodox medicine).

Quality evaluation instrument

The QEI was developed using NALFD guidelines released by the Korean Association for the Study of Liver in 2013, which evaluated the following knowledge domains of an NAFLD website: NAFLD definition, epidemiology, risk factors, disease natural history, comorbidities, complications, diagnosis, treatment modalities consistent with lifestyle modifications, and medical and surgical treatments (Table 1).

Table 1.

Description of the components of the quality evaluation instrument (QEI) used to evaluate websites with information on NAFLD (scores range from 0 to 56)

Defines NAFLD
				Describes NAFLD as term that includes non-alcoholic fatty liver, NASH and non-alcoholic fatty liver associated cirrhosis (1)
				Define significant alcohol intake amount differentiate from alcoholic liver disease (2)
Epidemiology of NAFLD
		Discusses incidence and prevalence of NAFLD
				Incidence (3)
				Prevalence (4)
				Discusses risk factor of NAFLD
				Obesity (5)
				Diabetes mellitus type II (6)
				Dyslipidemia (7)
				Prevalence (4)
				Metabolic syndrome (8)
				Hypothyroidism (9)
				Polycystic ovarian syndrome (10)
				Sleep apnea (11)
Discusses natural course of NAFLD
				Total mortality (12)
				Liver related disease progression (13)
				Cardiovascular disease as most common cause of mortality (14)
Discusses comorbidity of NAFLD
				Cardiovascular disease (15)
				Diabetes mellitus type II (16)
Diagnosis of NAFLD
		Discusses about when and which test should be performed to diagnosis and screen of NAFLD.
				Discusses about diagnostic method
						History taking and serologic test for rule out other cause (17)
						Blood liver function test (18)
						Abdominal sonography (19)
				Discusses screening indication
						Insulin resistance patient (20)
						Blood test and abdominal sonography (21)
				Discusses comorbidity screening in NAFLD patient
						Metabolic syndrome (22)
						Cardiovascular disease (23)
						Diabetes mellitus type II (24)
		Discusses non-invasive test about NASH and advanced fibrosis
				Radiology method
						Liver ultrasonography (25)
						Computed tomography (26)
						Magnetic resonance image (27)
						Magnetic resonance spectroscopy (28)
				Elastography
						Transient elastography (29)
						Controlled attenuation parameter (30)
						Magnetic resonance elastography (31)
				Biochemical panel
						NAFLD fibrosis score (32)
						Enhanced liver fibrosis panel (33)
						Cytokeratin-18 fragments (34)
		Discusses liver biopsy indication and efficacy with complication
						Indication (35)
						Efficacy (36)
						Complication (37)
Management of NAFLD
		Discusses treatment modality of NAFLD
				Lifestyle modification: discusses efficacy and method
						Weight reduction (38)
						Diet (39)
						Exercise (40)
						Drug for weight reduction (41)
				Medical treatment: discusses efficacy and complication
						Anti-oxidant
								Vitamin E (42)
								Other anti-oxidant (43)
						Insulin sensitizer
								Thiazolidinediones (44)
								Metformin (45)
						Lipid lower agents
								Statins (46)
								Fibrates (47)
								Omega-3 polyunsaturated fatty acid supplementation (n-3 PUFAs) (48)
								Ezetimibe (49)
						Pentoxifylline (50)
						Ursodeoxycholic acid (UDCA) (51)
						Angiotensin-II receptor blockers (52)
						Investigating others drugs (53)
		Surgical treatment: discusses indication, efficacy, and complication
						Bariatric surgery
								Indication (54)
								Efficacy (55)
								Complication (56)

NAFLD, non-alcoholic fatty liver disease; NASH, nonalcoholic steatohepatitis.

Each domain consisted of several relevant items. If an item was not discussed or named in the website or if the discussed information was incorrect, no points were given for that item. The QEI score is expressed by the sum of the points from individual items. The possible score ranged from 0 to 56.

Global quality score

The GQS is a tool for the evaluation of the overall quality of the site, which includes the flow of information, ease of use, and usefulness for patients. It uses a 5-point Likert scale to rate the overall quality of the site (Table 2).

Table 2.

Global quality score criteria used to score the websites with information on non-alcoholic fatty liver disease

Global score	Description
1	Poor quality, poor flow of the site, most information missing, not at all useful for patients
2	Generally poor quality and poor flow, some information listed but many important topics missing, of very limited use to patients
3	Moderate quality, suboptimal flow, some important information is adequately discussed but others poorly discussed, somewhat useful for patients
4	Good quality and generally good flow, most of the relevant information is listed, but some topics not covered, useful for patients
5	Excellent quality and excellent flow, very useful for patients

DISCERN instrument

The DISCERN instrument was developed to evaluate written health information about treatment choices available to patients and healthcare professionals. The benefit of the DISCERN instrument is that it is possible to evaluate the quality of a publication without the need for expert knowledge and without reference to other publications or advisers [25]. It consists of 16 questions and is divided into three sections: first, a section about the reliability of the publication (questions 1–8), second; a section about the quality of information regarding treatment choices (questions 9–15); and third, a section about the overall quality of the informational material (question 16). The 16th question was not used as it is essentially the same as the GQS. The first 15 questions scored using a 5-point scale, where 1 is “definitely no,” 2–4 represent “partially yes,” and 5 is “definitely yes” [26].

Statistical analysis

All analyses of collected data were conducted using SPSS software version 21.0 (IBM Co., Armonk, NY, USA). For the data collected by measuring website QEI, DISCERN, and GQS scores, the mean ± standard deviation, median, and interquartile range were calculated. The differences in QEI, GQS, and total DISCERN scores between groups were calculated by using non-parametric test method, the Kruskal-Wallis test. A p < 0.05 was considered significant. The Pearson correlation relationship and p values for significance were calculated to evaluate the relationship between the QEI and GQS. A p < 0.05 was considered significant. The degree of inter-rater agreement was evaluated by using the intraclass score for QEI, GQS, and total DISCERN score.

RESULTS

The searches using the three search engines, Naver, Daum, and Google, resulted in the identification of 332 websites. After removal of duplicates, we identified 206 websites. After applying the inclusion and exclusion criteria, 47 websites were finally identified and evaluated in the study (Fig. 1).

Figure 1.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flowchart showing the websites on non-alcoholic fatty liver disease (NAFLD) searched on the internet and those finally included in the study.

The overall QEI scores were low and varied widely, with a minimum score of 4 and a maximum score of 22. The mean score with standard deviation was 10.31 ± 5.09 (n = 47). Seventeen websites had scores above the mean. The median QEI score was 8, with a first and third quartile (Q1, Q3) of 6 and 15, respectively. Of the items, the definition, prevalence, risk factors, and lifestyle modification items scored relatively high, whereas the comorbidity, comorbidity screening, medical treatment, and surgical treatment items had low scores.

As with QEI scores, the total DISCERN score was poor and varied widely, with a minimum score of 18 and a maximum score of 43. The mean score with standard deviation was 24.66 ± 5.54 (n = 47), indicating poor information quality. The median total DISCERN score was 23, with a first and third quartile (Q1, Q3) of 21 and 27, respectively. The mean score for each question from question 1 to question 15 was less than 3, and the lowest scoring question was “Is it clear when the information used or reported in the publication was produced?” as related to the dates associated with the sources of evidence. The question with the highest mean score was “Does it describe what would happen if no treatment is used?” with regards to the risk or benefit associated with any no-treatment option (Fig. 2).

Figure 2.

Mean scores for the questions using the DISCERN scale, across the websites evaluated.

The median global quality score for all websites evaluated (n = 47) was 2.0. GQS scores of 1, 2, and 3 were applied to 7, 30, and 10 websites, respectively, indicating generally poor quality of information. There was no website that scored 4 or above according to the GQS. The relationship between the QEI and GQS was evaluated by calculating the Pearson correlation coefficient. There was a high correlation between the QEI and GQS (r = 0.74, p < 0.01) (Fig. 3).

Figure 3.

Relationship between the global quality score (GQS) and the quality evaluation instrument (QEI) score for evaluating non-alcoholic fatty liver disease websites. The QEI was highly associated with GQS (r = 0.74, p < 0.01).

The 47 websites were grouped into five categories based on the website creators, and the information quality for each of the groups are shown in Table 3. Of the evaluated websites, 32% were institutional, 45% were non-pharmaceutical commercial, 0% were pharmaceutical commercial, 6% were charitable, 17% were support, and 0% were alternative medicine. For all the groups, the quality of the websites was poor by the three evaluation measures. There was no significant difference in the QEI, GQS, and total DISCERN score between the groups (Table 3).

Table 3.

Grouping the websites on non-alcoholic fatty liver disease included in the study

Catergory	No. of websites (%)	QEI		Total DISCERN score		GQS
		Mean±SD[a]	95% CI for means	Mean±SD[b]	95% CI for means	Mean±SD[c]	95%CI for means
Institutional[d]	15 (32)	11.60 ± 4.80	8.93–14.26	26.20 ± 5.26	23.28–29.11	2.13 ± 0.35	1.93 to 2.32
Nonpharmaceutical commercial[e]	21 (45)	9.81 ± 4.98	7.54–12.07	25.95 ± 6.35	23.06–28.84	2.09 ± 0.62	1.81 to 2.38
Charitable[f]	3 (6)	5.33 ± 1.15	2.46–8.20	20.00 ± 2.00	15.03–24.96	1.33 ± 0.57	–0.10 to 2.76
Support[g]	8 (17)	11.12 ± 6.12	6.00–16.2	23.37 ± 5.92	18.42–28.33	2.12 ± 0.83	1.42 to 2.82
Total	47 (100)	10.31 ± 5.09	9.02–11.89	25.21 ± 5.86	23.49–26.93	2.06 ± 0.60	1.89 to 2.23

QEI, quality evaluation instrument; GQS, global quality score; SD, standard deviation; CI, confidence interval.

The QEI scores were not significantly different between the groups as per Kruskal-Wallis test (p = 0.058).

The total DISCERN scores were not significantly different between the groups as per Kruskal-Wallis test (combined, p = 0.211).

The GQS scores were not significantly different between the groups as per Kruskal-Wallis test (combined, p = 0.60).

This category includes government, hospital, or university.

This category includes sponsored sites or private medical sites.

This category include a nonprofit organization.

This category includes personal web pages or patient support groups.

Agreement between reviewers

Inter-rater agreement was assessed by using the intraclass score for QEI, GQS, and the total DISCERN score. The intraclass scores were 0.85 (p < 0.01), 0.84 (p < 0.01), and 0.89 (p < 0.01), respectively, indicating an acceptable agreement between the two reviewers (I.T.S. and H.I.J.).

DISCUSSION

The patients’ need to obtain information, including that regarding the cause, diagnosis, treatment, and prognosis of the disease, is growing. Because there is an unmet need for health information at outpatient clinics, many patients search the internet to obtain their health information [15,27]. The problem which patients encounter when searching the internet is not a lack of information, but rather the selection of adequate information among the overwhelming amount of internet sources. Appropriate health information from the internet must be reliable and evidence-based to ensure adequate quality.

Our study prospectively assessed the quality of NAFLD information on the internet using three search engines and three quality evaluation tools. Among the various tools for assessing the quality of health information, QEI, DISECRN scores and GQS are commonly used to assess the quality of health information for many diseases [20,21,28-30]. We found 332 possible websites through three search engines; however, only 47 websites met the enrollment criteria, and we found that the overall quality of the health information on the websites was poor.

The QEI results showed the marked variation and poor quality of NAFLD information. The overall QEI score was poor and the maximum QEI score was only 22 out of 56. Especially, there was little attention given to comorbidities, comorbidity screening, and medication or surgery as treatment methods. Recently, NAFLD has been regarded as a hepatic manifestation of metabolic syndrome. NALFD is an independent risk factor for cardiovascular disease and type 2 diabetes. The cause of the high mortality rate associated with NAFLD is thought to be cardiovascular disease rather than liver disease [6,31]. As such, the comorbidity and comorbidity screening items are important components of internet health information and need to be improved. The lack of medication and surgical treatment information is thought to be attributable to there being no definite, specific treatments except for lifestyle modification.

The GQS scores also indicated a poor quality of information, as most of the sites achieved scores of 1 or 2, with 10 sites achieving the score of 3, and no sites achieving scores of 4 or 5. There was a positive relationship between the QEI and GQS scores (r = 0.81, p < 0.01). Therefore, the QEI can be a tool for evaluating the quality of website information.

As with the two evaluation tools, the mean score for each DISCERN domain indicated a poor quality of information. The mean score for each DISCERN domain was less than 3, which indicates that the reliability and qualitative evaluation of the information about the treatment is not sufficient, and much improvement is needed.

The DISCERN instrument focuses on treatment information, while the QEI and GQS focus on broader aspects of the website information. The QEI and GQS are the methods of evaluation for use by health professionals, but the DISCERN instrument is a tool which can be used by patients. As such, there is a question about whether this tool is a valid measure when used by patients, but one study found a significant correlation between the DISCERN ratings of consumers, such as patients, and health professionals (r = 0.77, p < 0.01) [32].

Government, educational, and nonprofit organization websites are generally recommended as sources of high-quality, reliable health information [32]. However, our study demonstrated that there are no significant qualitative differences between websites created by institutional (e.g., government, hospital, or university), nonpharmaceutical commercial (e.g., sponsored sites or private medical sites), charitable (e.g., a nonprofit organization), and support (e.g., personal webpages or patient support groups) groups, and websites from all groups displayed poor quality. We speculate that the differences in criteria for ranking websites among the search engines may be the reason why the qualities of government, educational, and nonprofit organization websites were assessed to be poor and not significantly different from other groups. While most of the websites enrolled in this study were found using the Naver and Daum search engines, other studies of other countries’ assessments of internet health information quality were mostly performed using the Google and Yahoo search engines.

As shown in our quality evaluation results, our study found the quality of the internet information about NAFLD to be generally poor. In addition to our study, several studies of internet health information assessments for other diseases demonstrated marked variation in quality and poor quality information in need of many improvements, although these studies used different evaluation tools [21,28,33,34]. Therefore, there is great concern regarding the quality of internet health information [18,35].

Inadequate information quality results in misperception and changes in patient behavior regarding the disease, which can adversely affect the natural history and outcome of the disease. Comprehensive, unbiased, high-quality, and reliable internet health information can complement the information provided by healthcare professionals for a limited time, which can improve patient comprehension of the overall disease and communication between healthcare professionals and patients, as well as positively affect the disease course and treatment outcomes.

In order to use internet health information in a way that helps patients, it may be possible for healthcare professionals to suggest a website with the appropriate quality of information to the patient, but an accreditation system is also needed allow patients to identify a website of a certain quality in the milieu of constantly changing internet information. Therefore, efforts should be made to implement an accreditation system developed by a group of experts, including academic societies, in order to identify websites with an appropriate quality of information for patients. Additionally, there is a need for institutional support and some degree of information regulation.

Our study does have several limitations. First, we did not implement a readability assessment as part of the information quality evaluation tool. Even if the health information content is excellent, it is important to provide information that can be read and understood by the general public and patients. Generally, information for the general public is recommended to be written at a reading level of 11 to 14 years of age [36,37]. The Flesch Reading Ease score and the Flesch-Kincaid grade level score in the Microsoft Word program are the most commonly used tools for evaluating the readability of information in English. However, there is currently no tool to evaluate the readability of information in Korean.

Second, we were unable to analyze the correlation between the information quality rank and website rank. In a study evaluating the quality of inflammatory bowel disease information using the Google search engine, there was no relation between search rank and quality rank [20,30]. Of the 47 enrolled websites in our study, 43 websites were found using Naver and Daum. Naver and Daum search engines, unlike Google, do not list websites sequentially, but instead list them in categories, such as encyclopedia, blog, cafe, etc. Therefore, we could not analyze the correlation between website rank and information quality rank.

Third, websites enrolled for evaluation were selected on a specific day using the specific search term “non-alcoholic fatty liver disease” in Korean. Internet information can be constantly updated and changed, and there can be qualitatively good websites in languages other than Korean. Thus, websites that would meet our enrollment criteria may vary depending on the time and language of the search. In other studies, however, quality was assessed by searching for specific periods and words [21,29,30,34].

Finally, although this study showed that the correlation between the two reviewers was good, the reliability of these two reviewers may be questioned, specifically with regards to how representative and objective they were. Therefore, more reviewers need to participate in the evaluation, and it is necessary to assess the differences between evaluators.

Despite these limitations, this study had notable findings. To our knowledge, ours is the first study to evaluate NAFLD information on internet using widely accepted, validated tools and a QEI which was developed based on the Korea NALFD guidelines. We also analyzed the correlation between QEI and DISCERN scores, suggesting that the QEI is also suitable for quality evaluation.

The use of the internet as a source of health information is steadily increasing. The information from the internet influences patient knowledge, attitudes toward disease, and disease course and outcome. Our study highlights that the quality of information about NAFLD available on the internet was overall poor and in need of much improvement. Therefore, to support development of an accreditation system to provide patients with internet health information of appropriate quality, institutional support, and qualitative regulation of internet information are needed.

1. The quality of non-alcoholic fatty liver disease information resources on the internet was overall poor and in need of much improvement.

2. We need development of an accreditation system, institutional support, and qualitative regulation in order to provide patients with appropriate quality internet information.