What Factors Impact Consumer Perception of the Effectiveness of Health Information Sites? An Investigation of the Korean National Health Information Portal.

Lay public's concerns around health and health information are increasing. In response, governments and government agencies are establishing websites to address such concerns and improve health literacy by providing better access to validated health information. Since 2011, the Korean government has constructed the National Health Information Portal (NHIP) website run in collaboration with the Korean Academy of Medical Sciences (KAMS). This study therefore aimed to 1) examine consumer use of NHIP, with respect to the usage patterns, evaluation on health information provided, and perceived effectiveness of the site; and 2) identify factors that may impact perceived effectiveness of the site. An online survey was conducted with 164 NHIP users, recruited through a popup window on the main screen of the portal website from October to November 2015. The significant predicting factors supported by the data include the relevance of health information on the site, the usefulness of information in making health decisions, and the effective visualization of information. These factors can inform future efforts to design more effective health information websites, possibly based on metadata systems, to further advance the lay public's information seeking and health literacy.

INTRODUCTION

Due to aging and increasing chronic illness, the importance of capacity-building for disease prevention and self-management is increasing (1). Health information is a key element in strengthening individuals' ability to manage their own health, and health literacy is what allows them to find and evaluate health information themselves, actively and not as mere passive consumers, and to build more egalitarian, cooperative relationships with healthcare providers through health information sharing (2).

Accordingly, there is an increasing demand that health information be provided conveniently through a variety of media including the Internet. However, there is a high chance that people are being flooded with unverified health information and are becoming confused, making decisions that negatively affect their health (3). Therefore, national health systems should be considering how to provide health information that can be easily found by the entire population. For this reason, many governments and health agencies are disseminating health information via the Internet and trying to ensure that it is verified and easy for laypeople to understand.

For example, in the United States, the Centers for Disease Control and Prevention (CDC) provide a variety of information through their website, and the National Library of Medicine (part of the National Institutes of Health) operates MedlinePlus, which links health information from over 100 institutions and provides with linked 35,000 health information. Also, federal, state, and local governments provide health information services through NHS Choices in UK and Healthdirect in Australia (3).

In Korea, the Ministry of Health and Welfare has operated since 2011 a governmental health information portal site (http://health.mw.go.kr), in cooperation with the Korean Academy of Medical Sciences (KAMS). In particular, in 2007 the KAMS began to be in charge of developing a wide range of validated health information and contents on behalf of the Ministry, and in 2011 the National Health Information Portal (NHIP) website was launched.

The NHIP is meant to provide the public with health information that is verifiable and easy to understand. In more detail, this means it should be scientifically sound and up-to-date (which is the responsibility of the Korean Medical Association and 160 medical professionals' societies); visually enriched (with images and videos) and repurposable (for example, sharable on a user's blog by request).

The use of health information online is increasing and expected to continue to increase; however, if health information is provided based only on the perspectives of experts, bureaucrats, and/or web developers, it might not meet the needs of users. For effective health information provision online, it is necessary to pay attention to the opinions and evaluations of portal users and their experiences using the healthcare system. Such research is already ongoing in many countries (3). In Korea, the number of visits to the NHIP is more than 2 million per month and expected to rise. However, there has been no effort to investigate the effectiveness of the portals for citizens, only suppliers (4).

The goals of this study are 1) to examine consumer use of the NHIP, focusing on their usage patterns, evaluation on health information provided on the site, and perceived effectiveness of the site; and 2) more importantly, to identify important factors affecting the perceived effectiveness of the site. The findings of this study will have practical implications in guiding future site development and research in the domain of health information engagement and literacy.

MATERIALS AND METHODS

Participants

The results of this study were obtained from a panel survey constructed to evaluate the effectiveness of the NHIP for users and confirm factors affecting effectiveness. The panel was composed of users who voluntarily agreed to participate through a popup window on the portal homepage. A total of 164 respondents agreed to participate; all were included. Participants provided informed consent and were paid US $30 each for participating. The size of the required sample was determined by a formula using the population size, confidence level, and error range. The average number of visitors per month in the NHIP is 20,000. When the questionnaire is conducted with a probability of 10% with 95% confidence, it is necessary to get over 100 persons on the sample size. People who participated in the panel is considered as a representative sample (5).

The survey period stretched from October 26, 2015 to November 30, 2015; the survey was conducted online using SurveyMonkey (San Mateo, CA, USA).

Survey content

The survey covered the demographic characteristics of respondents, their usage patterns of the NHIP site, evaluation of the health information available on the site, and perceived effectiveness of their site use. The demographic characteristics covered were gender, age (in decades), and education level (high school graduate only and university/college graduation or higher).

The patterns of NHIP usage was examined by assessing 1) the sources of awareness, such that how come participants first learned about the portal; 2) the type of health information they looked for on the portal; and 3) the frequency of portal use over the past year.

The evaluation of health information provided on the NHIP was assessed using a set of 13 items asking participants' satisfaction with various aspects of health information. For the evaluation questionnaires, some items were added into Evaluation Checklist for Internet Health/Disease Information (ECIHD), 8 items, which are developed to evaluate the quality of domestic health information by Kim et al. (6). Additional items were 1) whether the information helped to make personal decisions (3 items); and 2) whether data visualization was effective (2 items).

Lastly, the perceived effectiveness of the NHIP was examined in terms of 1) participants' overall satisfaction with their portal site use; 2) intention to revisit the site; and 3) intention to recommend the site to others. Scoring of survey 13 items ranged from “very much” to “not at all” on a 5-point Likert-type scale.

Analysis methods

Frequency analysis was performed to identify the population's demographic characteristics. For portal usage patterns, the sources of portal awareness (Fig. 1) and the type of information searched were calculated by considering multiple responses; the frequency of portal use was calculated as the frequency of access in the last year.

Fig. 1

Result of awareness of NHIP.

NHIP = National Health Information Portal.

For evaluation on the information available on the portal, exploratory factor analysis (EFA) was used to determine the construct validity of the items. In EFA, the researcher has no particular expectations about the number and nature of factors; principal component analysis with varimax rotation is the most commonly used method to find factors (7).

Based on the results of the factor analysis, the communality, which indicates the amount of information related to the variables included in the extracted factors, was analyzed, and items with scores of less than 0.5 were deleted. The final number of final factors was selected based on an eigenvalue of 1 and a cumulative variance ratio over 0.6. Cronbach's alpha was calculated to determine the internal consistency, which was commonly used to estimate the reliability of the questionnaire.

For effectiveness of the health information portal, Likert-type results for overall satisfaction, intention to revisit, and intention to recommend to others were broken into “high” and “low” by median scores.

Because the 3 items on the effectiveness of the health information portal were dichotomous (“high/low”) variables, multiple logistic regression analysis was used and the odds ratio (OR) of effectiveness of the health information portal was verified by a 1 score increase of each item for health information evaluation. The collected data were analyzed using SPSS ver. 23 (IBM Corp., Armonk, NY, USA). All statistical tests were 2-sided. Statistical significance was set at a P value < 0.05.

Ethics statement

The present study protocol was reviewed and approved by the Institutional Review Board of Kangwon National University (KWNUIRB-2015-08-002). Informed consent was submitted by all subjects when they were enrolled.

RESULTS

The sociodemographic characteristics of the surveyed population

Among the 164 respondents, men accounted for 23.8% and women 76.2%. As for age, people in their 20s accounted for 31.7%, 30s for 35.4%, 40s for 14.0%, and 50s and over for 18.9%. On education level, the proportion of “high school graduate only” was 27.4% and that of “university (including college) graduation or higher” was 72.6% (Table 1).

Table 1

The sociodemographic characteristics of the surveyed population

Characteristics	No. of persons
Total	164 (100.0)
Sex
Male	39 (23.8)
Female	125 (76.2)
Age group
10–29	52 (31.7)
30–39	58 (35.4)
40–49	23 (14.0)
≥ 50	31 (18.9)
Education level
High school graduate only	45 (27.4)
University (including college) graduation or higher	119 (72.6)

Values are presented as number (%).

Usage patterns in relation to the NHIP

Awareness of NHIP (multiple responses)

Forty-six point two percent of the respondents answered that they were “aware through searching of a keyword related to health information in the portal site search engine.” The source of this awareness could be broken down into “awareness through others” (43.6%), “awareness through Internet use other than portal site search” (26.3%), “awareness through medical personnel” (11.5%), and “awareness through mass media” (5.1%) (Fig. 1).

Type of information searched through the NHIP (multiple responses)

“The characteristics of disease/symptoms” accounted for 62.7%, the highest. “Treatment and management methods” accounted for 46.8%, “improvement in lifestyle (exercise, nutrition, drinking, and non-smoking)” for 39.9% “preventive measures” for 39.2%, “diagnostic methods” for 38.6%, and “medical institutions and medical personnel” for 10.8%—the lowest. Other responses included “basic medical information,” “reference materials,” “human anatomy,” and so on.

Frequency of NHIP use

The frequency of accessing the NHIP in the past year was 38.4% for “once (this connection is the only one),” 16.5% for “less than once a month,” 12.2% for “once a month,” 17.7% for “2 to 4 times a month,” 10.4% for “2–3 times a week,” and 4.9% for “more than 4 times a week.”

Site users' evaluation about health information on the NHIP: EFA

EFAs revealed 4 factors with an eigenvalue > 1, explaining 63.51% of the total variance (Table 2). The Kaiser-Meyer-Olkin (KMO) index in this study was 0.825, indicating sufficient data for analysis. Bartlett's test of sphericity, with 655.514, was also significant (P value < 0.001), indicating that the correlations between items were sufficient and warranted factor analysis (89).

Table 2

EFA and reliability analysis of health information evaluation

Factor	Item	Factor loading				Cronbach's alpha
		Factor 1	Factor 2	Factor 3	Factor 4
Relevance of health information on the site	The content of the information is easy to understand.	0.760	-	-	-	0.777
	The content of the information is generally beneficial.	0.702	-	-	-
	The content of the information is appropriate for my level.	0.688	-	-	-
	The information is suitable for the public situation.	0.601	-	-	-
	The information contains enough of what I need.	0.556	-	-	-
Usefulness of information in making health decisions	The information is sufficiently detailed.	-	0.765	-	-	0.724
	The length of the information is appropriate.	-	0.650	-	-
	The information helps me choose prevention/diagnosis/treatment methods.	-	0.626	-	-
	The information is timely/appropriate.	-	0.560	-	-
Comprehensiveness of information	It shows us how to get additional information or ask for help when using health information.	-	-	0.810	-	0.620
	It provides precautions for use of the information.	-	-	0.743	-
Effective visualization of information	Visual data help convey the information.	-	-	-	0.810	0.704
	Multimedia video resources help us understand the information.	-	-	-	0.788
Eigenvalue		2.488	2.332	1.817	1.619	-
Cumulative variance ratio (%)		19.142	37.081	51.059	63.512	-

EFA = exploratory factor analysis.

Two items, “The information is convenient to use” and “It's fun to find information on this website,” exhibited low communality scores (< 0.5) and were removed from the analysis. The minimum factor loading for each item in the factor matrix and the rotated matrix was 0.556.

As for factor analysis of health information evaluation items, 13 items were classified into 4 factors related to content: “the relevance of health information on the site,” “the usefulness of information in making health decisions,” “comprehensiveness of information,” and “effective visualization of information.” Cronbach's alphas for the 4 factors were 0.777, 0.724, 0.620, and 0.704, respectively, and showed high internal consistency.

Perceived effectiveness of the NHIP and influencing factors

The results for participants' perceived effectiveness of the health information portal, are “satisfied overall” at 71.3%, “intention to revisit” at 51.2%, and “intention to recommend to others” at 40.9%. We further examined what factors may impact the perceived effective of the NHIP (Table 3) by conducting a 3 sets of multiple logistic regression analysis. First, multiple logistic regression analysis with overall satisfaction as a dependent variable showed that the relevance of health information on the site and the usefulness of information in making health decisions were statistically significant—as they increased, so did overall satisfaction (ORs, 1.85 and 2.53, respectively), confirming that they positively affect overall satisfaction.

Table 3

Multiple logistic regression models of factors related to perceived effectiveness of the NHIP

Category (ref)	Perceived effectiveness of the health information portal
	Overall satisfaction (ref = low)		Intention to revisit (ref = low)		Intention to recommend to others (ref = low)
	OR	95% CI	OR	95% CI	OR	95% CI
Sex (male)		P = 0.363		P = 0.369		P = 0.089
Female	1.52	0.617–3.735	0.64	0.243–1.692	0.41	0.146–1.144
Age group (19–29)		P = 0.629		P = 0.044		P = 0.456
30–39	0.50	0.175–1.437	0.50	0.180–1.392	0.45	0.157–1.312
40–49	0.54	0.141–2.073	0.37	0.101–1.333	0.46	0.120–1.719
≥ 50	0.67	0.195–2.341	0.18	0.055–0.598	0.54	0.165–1.765
Education level (high school graduate only)		P = 0.701		P = 0.694		P = 0.031
University graduation or higher	0.83	0.311–2.193	0.82	0.312–2.171	3.04	1.107–8.357
Relevance of health information on the site		P = 0.004		P < 0.001		P < 0.001
	1.85	1.210–2.819	3.05	1.814–5.130	3.64	2.074–6.390
Usefulness of information in making health decisions		P < 0.001		P < 0.001		P < 0.001
	2.53	1.615–3.965	3.38	2.050–5.588	4.18	2.448–7.135
Comprehensiveness of information		P = 0.811		P = 0.550		P = 0.999
	1.05	0.705–1.563	0.88	0.586–1.328	1.00	0.657–1.523
Effective visualization of information		P = 0.585		P = 0.010		P = 0.016
	1.12	0.746–1.679	1.75	1.143–2.689	1.69	1.101–2.600

NHIP = National Health Information Portal, ref = reference level, OR = odds ratio, CI = confidence interval.

Second, with reuse intention as a dependent variable, we found that age, the relevance of health information on the site, the usefulness of information in making health decisions, and effective visualization of information were statistically significant. Compared with people in their 20s, above-50s had lower reuse (OR, 0.18); as relevance of health information on the site, usefulness of information in making health decisions, and effective visualization of information increased, reuse intent ORs were 3.05, 3.38, and 1.75, respectively.

Finally, with recommendation to others as a dependent variable, we found that education level, the relevance of health information on the site, the usefulness of information in making health decisions, and effective visualization of information were statistically significant. Compared with high school graduate only, the level of recommendation by people who were university graduates was higher (OR, 3.04), and as the relevance of health information on the site, usefulness of information in making health decisions, and effective visualization of information increased, recommendation level did also (ORs, 3.64, 4.18, and 1.69, respectively).

DISCUSSION

Our survey finding suggest people's demand for health information is rapidly increasing, and effective use of health information has been proven to help health behavior and health. According to the results of survey, patients who have access to appropriate health information and use it well are more likely to have better outcomes (10111213). However, the provision of commercial health information by private institutions has many problems in terms of reliability and quality of information. Also, according to the Health on the Net Foundation (HON, Geneva, Switzerland) survey 2015 (14), 79% of health system users answered that government and government agencies were responsible for online health information provision. Therefore, it is important for government to take responsibility to provide people with correct and appropriate health information (14).

Another responsibility of the government is sound management of health information. Currently, many health sites, especially commercial sites, contain unverified and in some cases incorrect information. However, it is difficult for laypeople to evaluate the quality of health information sites and evaluate whether information is commercial or not. Most people want to visit a site and easily find information, but few are interested in basic questions about who created this information and why (15).

Eysenbach and Köhler (13), using focus group, usability test, and in-depth interview results to study how to find information on health information sites, found that users searched the content that they came to find first, but only rarely searched about “about us.” Currently, there is no policy to control health information quality and avoid distortion led by government in Korea. It is necessary for the government to evaluate information providers, filter out wrong information, and identify websites with high quality and introduce them through the health information portal.

Various evaluation tools for evaluating online information have been developed, as have new standards and methods for developing and providing health information (6161718). However, these tools and criteria have been conducted from the perspectives of providers, not consumers, and the evaluation has often been performed by experts, not actual users. To increase the utilization of health information by lay people and improve the effectiveness of health portal sites, it is important to be aware which factors affect users' site satisfaction, intention of reuse, and intention to recommend to others.

Currently, several health sites are being operated by the Korean government and its agencies. In addition to the NHIP, there are “People In” provided by the National Health Insurance Corporation, health promotion information provided by the Korea Health Promotion Development Institute, and information on medical institutions provided by the Health Insurance Review & Assessment Service (HIRA). However, research has not previously been conducted to identify the advantages and disadvantages of each site from the user perspective, a gap this study aimed to help fill.

We found that the characteristics of health information content that make it relevant and aid treatment decisions affect overall satisfaction with the portal site, intention to reuse, and intention to recommend to others. In this sense, the results of this study are similar to those of previous studies; for instance, in the HON survey 2015 results, information relevance, indicating the extent to which the information answers the searcher's questions, was most important to health system users (14).

Eysenbach et al. (19) presented that most frequently used quality criteria included readability, visual images, and references provided through their results of systematic review on assessing the quality of health information for consumers. We also found that satisfaction with visual data positively affected site reuse and recommendation. Recently, visualization, including the use of infographics, has been emphasized as an information delivery method for big data.

We did not address search convenience in our research, but it is likely an important factor. Eysenbach et al. (19) have emphasized that it is important to improve the user's search experience on health information portals such as MedlinePlus, Mayo Clinic (Mayo Foundation for Medical Education and Research, Rochester, MN, USA), HON, and HealthInsite (Health|Insite (Pty) Ltd., Johannesburg, South Africa), and Better Health (Better Health, LLC., Miami, FL, USA). Another factor that should be considered is user experience, including customization, and satisfaction with search and information found. Finally, the matter of sharing metadata among health sites and portals across agencies and even countries should be considered.

The object of this questionnaire was recruited by non-probabilistic convenience sampling, meaning they may not representative of all users of the portal site. However, Ko and Lee (20) suggest that research based on non-random samples can provide useful qualitative as well as quantitative data. This study aimed to find factors affecting experience of the portal site as effective.

In conclusion, this study examined the information attributes that affect overall satisfaction with the site, willingness to reuse, and intention to recommend the portal to other people, from among a panel of users. The results showed that appropriateness of health information content and information characteristics that help decision-making affect all 3 dependent variables, while satisfaction with visual information positively affected reuse and recommendation intention. The results of this study will be used as basic data to establish strategies to develop better health information provision through the NHIP.