Using the Google™ Search Engine for Health Information: Is There a Problem? Case Study: Supplements for Cancer.

We assessed the quality of online health and nutrition information using a Google™ search on "supplements for cancer". Search results were scored using the Health Information Quality Index (HIQI), a quality-rating tool consisting of 12 objective criteria related to website domain, lack of commercial aspects, and authoritative nature of the health and nutrition information provided. Possible scores ranged from 0 (lowest) to 12 ("perfect" or highest quality). After eliminating irrelevant results, the remaining 160 search results had median and mean scores of 8. One-quarter of the results were of high quality (score of 10-12). There was no correlation between high-quality scores and early appearance in the sequence of search results, where results are presumably more visible. Also, 496 advertisements, over twice the number of search results, appeared. We conclude that the Google™ search engine may have shortcomings when used to obtain information on dietary supplements and cancer.

Introduction

Health professionals consult scholarly search engines like PubMed, Google Scholar™, the Cochrane Collaborative, and others to obtain high-quality information on health topics. Likewise, patients often turn to online searches in hopes of finding therapies after consulting with their physicians. In fact, 77% of Americans use a search engine to seek online resources about their disease, treatment, and alternatives on their own (1).

However, patients use different search engines, and the most popular is that of Google™, which may be contributing to the current health “misinfodemic.” Commercial search engines, like the Google™ search, are businesses. They are not primarily designed to be authoritative and unbiased sources of health-related information. The reliance on Google™ automated algorithms to regulate the quality and placement of information presented in search results is not a substitute for good clinical judgment. These algorithms are based on how well a search result achieves the purpose of Google™, which is to provide information that is most relevant for answering a user's query. The results may not be the most current, evidence-based, high-quality health information (2, 3). Although Google™ has an internal quality-rating system to continually improve its algorithms, its general guidelines and algorithms for rating and ranking webpage quality are proprietary and unavailable for scrutiny. Google™ claims its algorithms factor in internal standards for judging the expertise, authoritativeness, and trustworthiness (referred to as E-A-T) of website attributes, particularly for sites that contain information with potential impact on “the future happiness, health, financial stability, or safety of users,” which Google™ designates as “Your Money or Your Life” (YMYL) related (4). Such sites include those that provide advice and sell products related to diet, nutrition, and medical devices. Google™ employs search quality raters who grade how well the Google™ algorithms identify and make YMYL content with high E-A-T more visible to search users by ranking them highly, but neither the expertise of the raters nor the results are in the public domain.

Misinformation and disinformation sources, such as viral websites with spurious content and false health claims, are difficult for automated algorithms to spot. Unscrupulous web developers circumvent them by finding ways to make their webpages appear to be authoritative and trustworthy, even when they are not. The algorithms are unlikely to be sufficient to replace human expertise in screening for unverified health information. Therefore, additional scrutiny is necessary to identify high-quality results.

A recent systematic review involving 153 studies concluded that highly scoring sites were often unsuitable for lay users and were deficient in readability (5). The 2 most popular tools were the Health on the Net Foundation Code of Conduct (Hon Code) and DISCERN, but they had limitations. For example, the Hon Code consists of voluntary declarations by the website publisher that the website adheres to certain standards, but does not include objective review and validation by external observers to guarantee that the criteria have been met. DISCERN (http://www.discern.org.uk/discern_instrument.php), developed in 1999 by British investigators, is the most frequently used tool for screening consumer health information on medical treatments (5–7). It addresses some aspects of quality, but the questionnaire is >20 y old. It was not designed for evaluating online health information webpages and does not address Google™ E-A-T indicators, such as authorship and date of publication, which influence Search Engine Results Page (SERP) ranking.

Although the literature is still sparse, there is reason to suspect that online searches for health information have considerable limitations. A study of 10 webpages from a search for “scoliosis” was scored by 2 physicians for the comprehensiveness of search content, likeliness to be recommended to patients, and readability (8). The mean overall score was 47.6 out of 75 points. Physician rankings of webpage quality and their order of SERP appearance were inconsistent, and were inappropriately high, above Joint Commission recommendations for patient education materials to be written at or below a fifth-grade level (9).

In another evaluation of the accuracy, quality, and readability of 46 search results from 3 different search engines about the treatment of pediatric hypospadias, institutional websites scored significantly higher on accuracy than did commercial sites, but the readability scores were high (eighth grade) for all of the website categories. The mean quality score was mediocre: 57.5 out of 75 points using DISCERN (10).

Recently, Dobbins et al. (7) assessed the reliability and user-friendliness of health information screening tools, including DISCERN, and concluded that there was no screening tool designed for lay users that had <15 criteria. The few studies that have been conducted of online searches also suggest that easy-to-use screening tools for quality are needed.

The lack of appropriate tools has clinical implications. For example, dietary supplements are widely advertised and sold online as low-cost, patient-controlled therapies to “boost the immune system,” “provide more energy,” and “help prevent cancer,” although expert groups agree that there is no evidence that dietary supplements prevent or cure cancer (11–18). Nevertheless, many cancer patients use at least 1, and often many, dietary supplements to complement their treatment or, more rarely, to substitute for conventional cancer treatment (19, 20). For example, in the Breast Cancer Quality of Care (BQUAL) Study, a multicenter, prospective cohort study in women with breast cancer, 70% of respondents at baseline used dietary supplements and 87% used at least 1 complementary and alternative medicine (21). Supplement use was higher among the women who did not initiate chemotherapy than among those who did (88% vs. 62%). Of the 2272 adult cancer survivors and 31,310 adults without various cancers who participated in the NHANES 2003–2016, 70% of cancer survivors used any dietary supplement compared with 51% of healthy participants (22). Motivations included their attempts to manage medication side effects, provide emotional support, help themselves when conventional medicine seems unsatisfactory, and provide health benefits (19, 23).

The goal of this case study was to evaluate the quality of Google™ search results for the query “supplements for cancer” using the Health Information Quality Index (HIQI) we developed. We hypothesized that results scoring highest in quality on the HIQI would emerge first and, thus, in the most prominent place within Google™ SERPs. Visible placement in SERPs would provide the searcher with the soundest information earlier, thus enhancing the likelihood of calling the viewer's attention to the highest-quality information. We also wished to assess the feasibility of identifying credible nutrition-related information using the HIQI.

Methods

The Health Information Quality Index (HIQI)

In order to overcome the limitations of prior tools and to provide useful and transparent information for the results of web searches, we developed the HIQI, which accounts for the E-A-T elements used by Google™, to aid in evaluating the quality of such Google™ searches. The HIQI rates the quality of search results with an overall score ranging from 0 (lowest quality) to 12 (highest quality). HIQI criteria are characteristics that can be objectively assessed from the search result, including website domain, authoritative nature of the health and nutrition information, and its lack of commercial aspects. Not all commercial aspects are immediately apparent. Some webpages that are clearly commercial in nature are designated as “Ad” by Google Adwords™. Publishers of these webpages pay Google™ to display their products or webpages in noticeable spots on an SERP to drive traffic to their website. However, other webpages that are not Google Adwords™ Ads also have commercial aspects. They arise from other characteristics that are detailed in . Thus, the degree to which the search results exhibited a lack of a commercial aspect, even after the ads were eliminated, was also included in the HIQI.

TABLE 1

2020 HIQI: Scoring for Each SERP of a Google Internet search

Subscore	Criterion	Points allotted and criteria for scoring
Type of website	Domain of website: What is the website's domain?	0 .com or .net  • and corresponds to a commercial site or network often used by businesses
		or
		• site is hosted by Wikimedia Foundation
		1 .com or .net and website is Hon Code or URAC certified
		2 .org, .gov, or .edu and content is a news media purpose (press release, public statement)
		3 .com or .net but
		• webpage consists of original, peer-reviewed, scholarly research as described in website “About” page
		or
		• website is owned by a health care or nonprofit organization
		or
		• .org, .gov, or .edu and website is owned by an academic, governmental, health care, or nonprofit organization as described in website “About” page
Lack of commercial aspects	Commercial sponsorship: Is the webpage content sponsored by a commercial entity, published for ecommerce, or focused on selling a product?	0 Yes1 No
	Commissioned links: Does the website receive commission for embedding links to products or services for sale?	0 Yes1 No
	Possible financial conflict of interest: Do the authors report a conflict of interest with a commercial entity?	0 Yes1 No
Authoritative nature of health and nutrition information	Authorship: Authorship provided and/or evidence information was medically reviewed	0 No author listed or byline is the organization's name and no indication it was medically reviewed
		1 Author listed is not a medical, health, or relevant science professional and webpage was not medically reviewed
		2 Author listed is not a medical, health, or relevant science professional, but
		• webpage was medically reviewed
		or
		• webpage is a transcript of an interview with a medical professional
		3 Author listed is a medical, health, or relevant science professional
	Timeliness of information: Is date the information posted/updated timely?	0 No date1 ≥ 5 years old 2 < 5 years old
	Information documentation: References to reputable scientific source for information provided?	0 No1 Yes, on a list of hyperlinks or references to scientifically reputable sources, at the end of the article

HIQI, Health Information Quality Index; Hon Code, Health on the Net Foundation Code of Conduct; SERP, Search Engine Results Page; URAC, Utilization Review Accreditation Commission.

Scoring system

A webpage receiving a higher HIQI score is considered more likely to be an authoritative source of health-related information and more trustworthy than one with a low HIQI score. Table 1 presents the scoring system that was used for each search result and the 3 subscores that were developed. The type of website subscore (1 item, assigned 0–3 points) consisted of the domain, a commonly used characteristic thought to indicate the quality of health information. The lack of commercial aspects subscore (3 items, assigned 1 point each) consisted of evidence of commercial sponsorship, commissioned links, and apparent conflict of interest. The authoritative nature of health and nutrition information subscore consisted of 3 items, including 3 points for authorship, 2 points for timeliness of information, and 1 point for information documentation. The methodology for developing the HIQI, scoring categories and additional details on the scoring system, a coding manual, and a glossary of terms related to Google™ searches are available (Supplemental Materials).

Search results

The computer's search history and cookies were cleared before beginning the search to ensure that past searches would not influence results. The search “dietary supplements for cancer” was conducted on 11 May 2020 by Rater 1. The Google™ search engine identified 299 million possible results and displayed 187 that the algorithms deemed to be most relevant. Twenty-seven search results contained duplicated content or were considered irrelevant to the query subject. These were eliminated. The remaining 160 results were scored. In addition to the search results, 496 Google Adsense™ advertisements (which are labeled as commercial advertisements paid for by an advertiser) were also displayed among the results but not counted by Google™ as search results. Ads were discarded and not rated using the HIQI since they were of clear commercial intent.

Reproducibility of scoring procedures

Of the 160 search results scored by Rater 1, 5 were randomly chosen and scored by Rater 2 to provide assurance that the criteria were objective and scored reliably. Of the 5 random sample webpages, 3 were assigned matching scores and 2 differed by 1 point.

Rater 1 found 17 additional webpages difficult to score because a characteristic could not be well categorized according to the HIQI criteria definitions. These were adjudicated between Raters 1 and 2 after consulting the Manual of Operations (Supplemental Materials). The score most closely conforming to it was recorded.

Statistical analysis

All data analysis was performed using Stata 16.0 (StataCorp) and Microsoft Office 16 Excel. In order to examine the relation between order of the search result in the SERPs and quality of information of search results scored with the HIQI, Spearman's rank order correlation coefficient was calculated. The relation between order of appearance in the SERPs and a high-quality search result score on the HIQI (score of 10–12) was also examined using a second Spearman's rank order correlation. The level for statistical significance was P < 0.05.

Results

The median and mean HIQI score for the 160 results was 8, with scores ranging from 0 to a perfect score of 12. Only 11 of the results had perfect HIQI scores of 12; 7 were reprints of peer-reviewed publications and 4 were published on health care center and health information websites.

For descriptive purposes, we categorized total HIQI scores of 0–3 as poor, 4–6 as low, 7–9 as moderate, and 10–12 as high quality. Of the 160 search results, 9% (n = 14) scored poor, 31% (n = 50) scored low, 34% (n = 55) scored moderate, and 26% (n = 41) scored high.  presents the total HIQI scores and contributions of the 3 subscores (type of domain, lack of commercial aspects, and authoritative nature of the information) to the total scores. Details about each search result and their HIQI scores are available (Supplemental Table 1).

FIGURE 1

Mean scores and subscores by category for 160 Google™ search results for the query “supplements for cancer” using the 2020 HIQI. HIQI, Health Information Quality Index.

High-quality results, particularly the 26% receiving the highest scores of 10–12, rated highly on both the lack of commercial aspects and the authoritative nature of the health and nutrition information subscores. Many of the 34% webpages of moderate quality (scores 7–9) were published by major health care institutions. The remaining 40% of results (n = 64) had scores of ≤6 and were judged to be of low or poor quality. Results with a commercial emphasis (low lack of commercial aspects subscore) clustered primarily in the poor-quality category (0–3). Links to commercial entities indicated that some of those website publishers may have received revenue for referencing a product or incorporating affiliate links to commercial sites and merchandise.

The major shortcoming of low subscores on the authoritative nature of the health and nutrition information was lack of documentation; over half of the search results failed to disclose sources and references adequately. Referencing practices also varied greatly—from none, to hyperlinked texts, to display of formal bibliographies.

The correlation between a webpage's HIQI score and its rank order of appearance in Google™ search results was positive but not statistically significant (Spearman's ρ = 0.034, P = 0.67). We also examined the correlation between rank order of appearance in search results, focusing solely on webpages with high-quality HIQI scores (≥10; n = 41). The correlation was negative but, again, not statistically significant (ρ = −0.245, P = 0.12), indicating there was no apparent trend for higher HIQI scoring results being displayed earlier in the SERPs.

Results from the website domain subscore revealed that the domains were often not useful in determining whether the site was noncommercial. For example, some individuals and organizations registered with the .org domain were actually marketers of products or commercial enterprises. Therefore, we reran the Spearman's rank order correlation after removing the domain subscore from the total score. The correlation between a webpage's adjusted score and its rank order of appearance was positive and statistically significant, suggesting that higher scores were displayed somewhat earlier in the SERPs, but the association accounted for very little (only ∼4%) of the total variation (Spearman's ρ = 0.21, P = 0.0075).

The quartiles were re-examined based on the “new” total score with 9 possible points after eliminating the domain subscore. When only the high score quartile was considered (adjusted score of 7–9 points), the correlation was slightly positive between the highest quartile and search results rank, but it was not statistically significant (ρ = 0.074, P = 0.61). This led us to conclude that there was not a high correlation between search result rank and webpages with high HIQI scores.

Discussion

Our case study results are consistent with prior findings about the mediocre quality of online health information and potential for lack of patient comprehension (24, 20–23, 25). We found many problems that individuals seeking to obtain authoritative and objective information about dietary supplements for cancer would encounter from a Google™ search. The sheer number of the 160 most potentially relevant results was daunting. Only about one-quarter of them were high in quality when assessed using the HIQI. Moreover, the highest-quality results were not displayed first or near the top of the results, where they would be more likely to attract the searcher's attention. The presence of the remaining three-quarters of the results of lower quality might obscure the visibility of high-quality content and confuse an individual searching for authoritative, unbiased information. The Google™ search engine does not appear to rank webpages based solely on the quality of health information provided. Our results showed a weak but nonsignificant correlation between search result quality and rank order in SERPs only when the domain score was eliminated. When the highest quality webpages were examined, the absence of the domain subscore had no effect. Finally, e-commerce–related characteristics, such as links to commercial sites or product mentions, appeared in many of the results, in addition to the very large number of advertisements. All of these aspects of search results make it difficult for patients to separate objective, evidence-based information about dietary supplements and their use in cancer from less-authoritative sources.

More research and ongoing evaluations of the quality of web-based information on medical and nutritional topics are needed. Google™ is by far the most widely used search engine in this country, and thus it is of particular concern, as a recent US Department of Justice lawsuit and report to the House of Representatives have shown (24). We were motivated to publish this case study to encourage others to replicate this work on other topics.

As this case study represents only an n of 1, additional research with Google™ and other search engines on a variety of health and nutrition topics is needed to confirm our finding that health and nutrition information is not prioritized according to authoritativeness and lack of commercial aspects. The task is complex and requires collaboration with digital information and technology experts. The algorithms used to generate Google™ search results are dynamic and may change over time. Results may also vary depending on how a search is completed (e.g., using typed text on a computer or mobile device, or by audio, using voice search-enabled devices) and the e-commerce displayed. Claims that a webpage is medically reviewed may also affect SERP rankings. By default, Google™ results today are tailored to the user's location, which limits the generalizability and replicability of findings from searches conducted in different locations; a filter that limited the influence of this was eliminated in 2016 (25, 26). Although search history and cookies were cleared before carrying out the search, it is unclear if Google™ stores information about the searcher using data from Gmail and other GoogleTM LLC products. Being logged into a Google™ account may influence search results if additional information that cannot be cleared is used to tailor search results. All of these factors merit further exploration.

The strengths of this case study are that it simulated an actual Google™ search of dietary “supplements for cancer” that might be performed by a patient or consumer using all 160 of the relevant search results displayed, rather than the selective reviews of the first 10 or 50 search results done in previous studies (8, 10, 27). The HIQI identified objective and publicly available characteristics on webpages to identify high-quality health information.

The limitations are that reader comprehension was not included in our evaluation and should be in future studies. Even a “perfect” score on the HIQI did not guarantee that the information presented was accessible and understandable by an audience of varying health literacy levels. Comprehension should be considered when evaluating online information in the future. One method would be to subject search results to a Flesch Reading Ease or similar tests, which are accessible through Microsoft Word, Good Calculators, and Readability Formulas, among other tools (9, 28, 29). Such tools are quick and user friendly, however, a decision was made against adding a readability subscore post-hoc due to the risk of compromising the integrity of the study’s scoring procedure. As observed, not all webpages declared the latest publication date. Potential changes to webpage information may have occurred since the original search was conducted, which would require us to re-conduct the study and re-score each search result. Due to limited funding and personnel, this was not feasible. Thus, further testing for inter-rater reliability and validation of the HIQI with a readability subscore is needed. Despite theise limitations, the study’s findings remain relevant and time-sensitive. The role of online information merits urgent attention because evidence-based, public health messaging during a “misinfodemic” and pandemic is more important than ever (30, 31). Also, an automated version would be helpful to clinicians who wish to identify information sources for their patients, public health professionals, and researchers alike, since hand scoring of search results for quality is very labor intensive.

In conclusion, using the Google™ search engine for obtaining health information may have problems. Our search on “supplements for cancer” produced an uncurated mass of information spread across dozens of pages in no particular order, including some seemingly authoritative advice combined with press releases, advertisements, anecdotal reports, and false health-related claims regarding disease prevention and cures. These are likely to contribute to misinformation and confusion that lead to patient skepticism of or nonadherence to conventional medical therapies, with possible adverse influences on prognosis. More evaluations of the quality and comprehension of web-based searches on medical and nutritional topics are warranted.

Click here for additional data file.