Quality Assessment of Online Complementary and Alternative Medicine Information Resources Relevant to Cancer.

BACKGROUND: Complementary and alternative medicine (CAM) is often used by cancer patients and survivors in the US. Many people turn to the internet as their first source of information. Health information seeking through the internet can be useful for patients to gain a better understanding of specific CAM treatments to discuss with their healthcare team, but only if the information is comprehensive, high quality, and reliable. The aim of this article is to examine the content, writing/vetting processes, and visibility of cancer CAM online informational resources.
METHODS: Online CAM resources were identified by Google and PubMed searches, literature reviews, and through sources listed on various websites. The websites were analyzed through a modified online health information evaluation tool, DISCERN (score range = 1-5). The website's features relevant to the quality assessment were described.
RESULTS: Eleven CAM websites were chosen for analysis. The DISCERN analysis showed a range of quality scores from 3.6 to 4.9. Lower DISCERN scores were generally due to deficiencies in describing the writing, editing, and updating processes. A lack of transparency with authorship and references was commonly present.
CONCLUSION: Cancer patients interested in CAM need unbiased, evidence-based, reliable, high-quality, easily accessible educational materials. Individuals should use the guidelines followed in this analysis (including DISCERN and Medline Plus) to find reliable sources. Website developers can use CAM Cancer (NAFKAM), Beyond Conventional Cancer Therapies, Memorial Sloan Kettering Cancer Center, breastcancer.org, Office of Dietary Supplements, National Center for Complementary and Integrative Health, and Cancer.gov as models for trustworthy content.

Introduction

Complementary and alternative medicine (CAM), a term that has generated much debate, refers to a diverse group of health practices and interventions that are not part of conventional Western medical practice and whose associated philosophies may be alien to the biomedical paradigm. The term holistic medicine is sometimes used as a synonym for CAM, but more specifically refers to an approach to medicine that attempts to manage health issues by comprehensively addressing the physical, mental and spiritual dimensions of illness. CAM therapies can fall into a variety of categories including, biofield/energy, exercise, manipulative and body-based, mind-body, nutrition/diet, pharmacological and biological, and spiritual therapies. A 2012 study estimated that approximately 79% of adult cancer survivors and 68% of cancer-free adults in the US had used a least one form of CAM in the prior 12 months.

To gain a better understanding of cancer and the available treatment options, many patients look for information from various sources, including online. The internet is a common source of information and its importance for seeking health information is increasing as more people use the internet regularly. In 2000, about 52% of US adults were active online, which increased to approximately 90% in 2019. The Health Information National Trends Survey from 2019 found that the most common initial health information source for US adults was the internet (72.7%), followed by healthcare professionals (16%), publications (4.3%), and friends/family/co-workers (4%). A study analyzing HINTS data found that the rate of internet health information seeking among U.S. cancer survivors, had increased to 69.2% in 2017 from 53.5% in 2011. A study analyzing the National Health Interview Survey (NHIS) data found an increase in general (63.2% to 70.8%) and health specific (46.8% to 52.2%) internet use for cancer survivors from 2013 to 2018. Dee et al also found a possible association between health specific internet use and dissatisfaction with healthcare among survivors. With the increase in individuals seeking information on the internet and the common use of CAM among cancer patients, it is crucial that reliable, comprehensible, and informative online CAM resources are available. Prior studies of CAM information resources found many websites contained unreliable or inaccurate information.[9 -11] Patients want access to reliable CAM resources but may not have the training/education to identify reliable resources or differentiate between high- and low-quality information online. Individuals without this knowledge are vulnerable to reading and relying on incorrect information that can impact treatment decisions and outcomes. The aim of this article is to assess aspects of the quality of commonly viewed cancer CAM online informational resources.

Methods

Online cancer CAM resources were identified through general Google searches and literature reviews found through PubMed searches (performed January-June 2020). Various search terms were used in the Google search bar, such as “Complementary and alternative medicine,” “Cancer complementary and alternative medicine,” “cancer complementary medicine,” “integrative oncology,” “Integrative medicine for cancer,” “CAM resources,” “CAM information,” and “Online resources for CAM.” Additional resources were found through websites identified on the initial Google searches. Other articles were identified from personal files (JDW).[10,11,13,14] An additional set of Google searches were conducted by 5 different people within National Cancer Institute’s (NCI) Office of Cancer Complementary and Alternative Medicine (OCCAM) using 7 CAM therapy terms frequently discussed in literature about cancer and CAM: cannabis, coenzyme Q 10, high-dose vitamin C, laetrile/amygdalin, medicinal mushrooms, acupuncture, and Reiki. These searches were completed between August-October 2020 and were performed to determine the visibility of the selected CAM online resources defined by the Google rank (ie, whether or not the resource appeared on the first 2 Google results pages for these common CAM terms). These Google ranks were also used to ensure popular cancer CAM resources were not missed for analysis. Only one researcher’s data (MS, Searcher #5 in Table 1) was used to present visibility in the results due to the similarities between all 5 searchers’ results. In addition to using Google rankings, visibility and usage were assessed through web traffic data that was collected by contacting each website and requesting monthly or yearly traffic data specifically for their CAM webpages.

Table 1.

CAM Topic Google Rank Data.

CAM term	Searcher	Cancer research UK	Breastcancer.org	BCCT	NAFKAM	Komen	MSKCC	Oncolink	Mayo Clinic	NCCIH	ODS	Natural medicines	NCI (PDQ)
Cannabis		Y	Y	Y	Y	Y	Y	Y	Y	Y	Y	Y	Y
	1	X	-	-	-	-	-	-	-	-	-	-	X
	2	X	-	-	-	-	-	-	-	-	-	-	X
	3	X	-	-	-	-	-	-	-	-	-	-	X
	4	X	-	-	-	-	-	-	-	-	-	-	X
	5(MS)	X	X	-	-	-	-	-	-	-	-	-	X
Coenzyme Q 10		N	Y	Y	Y	Y	Y	Y	Y	Y	Y	Y	Y
	1	X	X	-	-	X	X	-	-	-	-	-	X
	2	X	-	-	-	X	X	-	-	-	-	-	X
	3	-	X	-	-	X	X	-	X	-	-	-	X
	4	X	X	-	-	-	X	-	-	-	-	-	X
	5(MS)	X	X	-	-	X	X	-	-	-	-	-	X
Vitamin C		Y	Y	Y	Y	Y	Y	Y	Y	N	Y	Y	Y
	1	X	-	-	-	-	X	-	X	-	-	-	X
	2	X	-	-	-	-	-	-	X	-	-	-	X
	3	X	-	-	-	-	X	-	X	-	-	-	X
	4	X	-	-	-	-	X	-	X	-	-	-	X
	5(MS)	X	-	-	-	-	X	-	X	-	-	-	X
Laetrile/Amygdalin		Y	Y	N	Y	N	Y	Y	N	N	N	Y	Y
	1	X	X	-	-	-	X	-	-	-	-	-	X
	2	X	X	-	-	-	X	-	-	-	-	-	X
	3	X	X	-	-	-	X	-	-	-	-	-	X
	4	X	X	-	-	-	X	-	-	-	-	-	X
	5(MS)	X	X	-	-	-	X	-	-	-	-	-	X
Medicinal mushrooms		Y	Y	Y	Y	Y	Y	Y	N	N	Y	Y	Y
	1	X	X	-	-	X	X	-	-	-	-	-	X
	2	X	X	X	X	X	X	-	-	-	-	-	X
	3	X	X	X	X	X	X	-	-	-	-	-	X
	4	X	X	X	X	X	X	-	-	-	-	-	X
	5(MS)	X	X	X	X	X	X	-	-	-	-	-	X
Acupuncture		Y	Y	Y	Y	Y	Y	Y	Y	Y	N	Y	Y
	1	X	X	-	-	-	X	-	-	-	-	-	X
	2	-	-	-	-	-	X	-	-	-	-	-	X
	3	X	X	-	-	-	X	-	-	-	-	-	X
	4	X	X	-	-	-	X	-	-	-	-	-	X
	5(MS)	X	X	-	-	-	X	-	-	-	-	-	X
Reiki		Y	Y	Y	Y	Y	N	Y	N	Y	N	Y	N
	1	X	X	-	-	X	-	-	-	-	-	-	-
	2	X	X	-	-	X	-	-	-	-	-	-	-
	3	X	X	-	-	X	-	-	-	-	-	-	-
	4	X	X	-	X	X	-	-	-	-	-	-	-
	5(MS)	X	X	-	-	X	-	-	-	-	-	-	-

Abbreviations: “Y”, yes, the website does have information on this topic; “N”, no, the website does not have information on this topic; “X”, the website did appear on the first 2 Google results pages for that term. “-”, the website did not appear on the first 2 Google results pages for that term.

Websites were selected based on content and intended audience. Mostly public education resources targeting patients and laypeople were used. The resources were selected if they provided CAM educational material and descriptions of CAM therapies written in English. Cancer specific resources were preferred but some non-cancer resources were included that have extensive CAM information mentioning use with cancer. Each resource was chosen based on focus, content, organization type, etc. Several resources were not chosen because they lacked information on specific therapies. We also attempted to get various resource types, such as clinical centers, government agencies and charity organizations.

A modified version of a validated instrument, DISCERN was used to assess the quality and reliability of online health information resources. The DISCERN tool has 15 questions assessing the reliability and quality of online healthcare treatment information. Question 12 (Does it describe what would happen if no treatment is used?) was eliminated from our assessment due to its irrelevance for our purposes. Each question is analyzed by using a 5-point rating scale (1-5). The final overall quality score is an average of all individual scores. The final score ranges were grouped as follows: low (1), moderate (2-3), and high (4-5) quality. All data from the websites were collected and analyzed by one of the authors (MS). The initial data collection and content analysis occurred between February and June 2020, with minor subsequent modifications.

Results

The following 11 websites met the established criteria and were reviewed. Please see Table 2 for the DISCERN scores for each site and Table 1 for Google rank data.

Table 2.

DISCERN Scoring for Online CAM Information Resources.

Discern instrument data
Resource	Q1: Are the aims clear?	Q2: Does it achieve its aims?	Q3: Is it relevant?	Q4: Is it clear what sources of info were used to compile the publication?	Q5: Is it clear when the information used or reported in the publication was produced?	Q6: Is it balanced and unbiased?	Q7: Does it provide details of additional sources of support and info?	Q8: Does it refer to areas of uncertainty?	Q9: Does it describe how each treatment works?	Q10: Does it describe the benefits of each treatment?	Q11: Does it describe the risks of each treatment?	Q13: Does it describe how the treatment choices affect overall quality of life?	Q14: Is it clear that there may be more than one possible treatment choice?	Q15: Does it provide support for shared decision-making?	Final Score
Cancer Research UK	2	2	5	3	3	4	5	5	5	5	5	2	5	5	4.1
Breast Cancer .org	5	5	5	2	3	4	5	5	4	5	5	3	5	5	4.4
BCCT	5	5	5	5	3	5	5	5	5	5	5	5	5	5	4.9
CAM Cancer	4	5	5	5	4	5	2	5	5	5	5	3	5	4	4.5
Susan G Komen	2	2	5	4	1	4	3	5	5	5	5	4	5	5	3.9
MSKCC	5	5	5	3	3	4	2	5	5	5	5	3	5	5	4.4
Oncolink	3	3	4	3	3	4	5	5	5	4	5	2	5	5	4.1
Mayo Clinic	3	3	3	3	3	4	1	5	5	5	5	2	3	4	3.6
NCCIH	5	5	4	3	3	4	3	5	4	4	5	2	5	5	4.2
ODS	5	5	4	4	3	5	3	5	5	5	5	2	4	5	4.4
Natural Medicines	5	5	4	4	4	5	1	5	5	5	5	4	3	3	4.1

Cancer Focused Organizations

Cancer Research UK

https://www.cancerresearchuk.org/about-cancer/cancer-in-general/treatment/complementary-alternative-therapies

The DISCERN score was calculated to be 4.1.

Information

Cancer Research UK is a charity dedicated to cancer research, education, and advocacy. The website has a section on CAM cancer therapy with subsections about cost, finding a provider, research, questions for your doctor, and outside resources. About 45 therapy summaries are provided with a general description, why patients use, how to use, side effects, research, cost, and cautions. There are also sections on finding clinical trials, cancer chat forums, and a nurse helpline.

The Google search showed Cancer Research UK in the first 2 results pages for all 7 CAM topics.

Writing and vetting process

Cancer Research UK thoroughly describes their writing, editing, and updating process. New content is created by the Health Information and Patient Information Web teams. The teams write literature reviews following specific criteria for relevant, evidence-based sources. Drafts are edited by a manager, external specialist, and panel of lay reviewers. A list of the reviewers and their credentials is provided. These summaries are reviewed every 12 to 36 months and updated when necessary. A date is provided on every page for when the information was last reviewed. Evidence-based references are provided for most of the summaries.

BreastCancer.org

https://www.breastcancer.org/treatment/comp_med

The DISCERN score was calculated to be 4.4.

Breastcancer.org is a U.S.-based non-profit organization for breast cancer education on various topics like complementary and holistic medicine (CHM). The CHM page has subsections on benefits, safety, effectiveness, finding a practitioner, talking to your doctor, views on CHM, resource guides, and research news. The site has detailed descriptions of several complementary techniques: how it works, considerations, what to expect, research, and cost. This information is also available in Spanish.

The Google search showed BreastCancer.org in the first 2 results pages for 6/7 topics on which they have information.

Breastcancer.org states that the content is created by an editorial team and reviewed by an advisory board of medical experts, including some members that do CAM research or clinical work, which are listed on the website with names and credentials. The site does not state how often updates occur. Based on the dates provided, it appears updates are done frequently. Evidence-based references are not provided.

Beyond Conventional Cancer Therapies (BCCT)

https://bcct.ngo/

The DISCERN score was calculated to be 4.9.

BCCT is a U.S.-based non-profit organization that provides cancer CAM education for researchers, clinicians, patients, caregivers, family, and patient advocates. The section, “Where to begin,” has information about coping with cancer, managing emotions, living a healthy life, researching options, making decisions, finding trustworthy information, frequently asked questions, etc. Another section discusses various topics such as different complementary therapies, types of cancer care, diagnostic procedures, etc. There are over 90 summaries on “Health practices” and “Natural products, herbs, nutrients and supplements” and another about cancer symptoms, side effects, and useful complementary therapies. The site also contains blogs, personal and education stories, news articles, and a list of outside resources.

The Google search showed BCCT in the first 2 results pages for 1/6 CAM topics for which they have information.

BCCT states that a group of healthcare, cancer, and CAM advisors construct and review the information published on the website. A protocol for updating is not provided but evidence-based references, authors (for most pages), editors with names and credentials, and the date of the most recent update for all published material are provided.

CAM Cancer (NAFKAM)

http://cam-cancer.org/en

The DISCERN score was calculated to be 4.5.

CAM Cancer provides cancer CAM information targeted to health professionals. The site is funded by the European Commission and hosted by Norway’s National Research Center in Complementary and Alternative Medicine (NAFKAM). The website has 4 sections: evidence-based summaries, special topics, methodology, and aims and organization. The website has a list of evidence-based CAM summaries on dietary approaches, herbs, manipulative/body-based, mind-body interventions, and other CAM therapies for cancer. Each summary addresses topics such as what is it, does it work, safety, and evidence. German and Norwegian translations are available.

The Google search showed CAM Cancer in the first 2 results pages for 1/7 CAM topics for which they have information.

The CAM Cancer site offers guidelines describing the writing, reviewing, and editing processes. Each summary has different authors who must apply and be approved to write one. The process is run by an executive committee, senior editor, responsible editor, editorial board, and technical editor. The CAM Cancer Collaboration staff are listed on the website with their affiliations and credentials. The site specifies that the summaries are updated regularly, ideally once a year. Summaries are put into archives and taken down from the website if they are over 2 years old. Evidence-based references and the writing, editing, and authorship history are provided.

Susan G. Komen

https://ww5.komen.org/BreastCancer/ComplementaryTherapiesIntro.html

The DISCERN score was calculated to be 3.9.

Susan G. Komen is an organization focusing on research, public health, global outreach, and advocacy for breast cancer. There is a section of the website, under the title Survivorship Topics, with descriptions of complementary and integrative therapies. Every summary explains what the therapy is, effectiveness, how it works, safety, interactions, doses, and alternate names. Some additional subsections explain scientific evidence, talking to your healthcare provider, finding a practitioner, and clinical trials. Spanish translated summaries are available.

The Google search showed Komen in the first 2 results pages for 3/6 CAM topics for which they have information.

The website states the content is developed by breast cancer experts but doesn’t identify who writes and edits the content. It is mentioned that the information in the “about breast cancer” section is codeveloped with Harvard medical school faculty and Dana-Farber/Brigham and Women’s Cancer Center staff. The information is sourced from current research published in reputable, peer reviewed journals. Evidence-based references are not provided. After each summary, it is specified that the information is taken from the Natural Medicines database. Although it is not clearly stated, the information seems to be updated regularly.

Cancer Centers

Memorial Sloan Kettering Cancer Center (MSKCC)

https://www.mskcc.org/cancer-care/diagnosis-treatment/symptom-management/integrative-medicine

The DISCERN score was calculated to be 4.4.

MSKCC hosts the Bendheim Integrative Medicine Center. MSKCC’s website has an Integrative medicine page exploring various topics such as: general CAM information, herbs and dietary supplements information, contact information, research and clinical trials, and frequently asked questions. The “about herbs” section has summaries of about 275 herbs, botanicals, and other products/therapies. The patient-focused summaries discuss how it works, purported uses, warnings, side effects, etc. The healthcare professional version gives the scientific name, clinical evidence, purported uses, mechanisms of action, warnings, contraindications, adverse reactions, interactions, and doses. Similarly, there is a database with summaries on Mind-Body Therapies. The patient-focused descriptions contain background information, how it works, uses, side effects, providers, and where to get treatment.

The Google search showed MSKCC on the first 2 results pages for 5/6 CAM topics for which they have information.

The writing and editing process is not described. No authors or editors are listed. A list of evidence-based references is provided for each summary. It is not stated how often updates occur, but a date for the last update is provided. The pages visited for this project had been updated within the last 2 years.

Oncolink Penn Medicine

https://www.oncolink.org/cancer-treatment/complementary-and-alternative-medicine

The DISCERN score was calculated to be 4.1.

OncoLink is a Penn Medicine cancer information website providing education on cancer types, treatment options, risks, prevention, and support for patients and healthcare providers. Within the cancer treatments section, there is a subsection on cancer CAM, which includes: basics of integrative oncology, important terms, safety, talking to your healthcare team, and helpful resources. The section on dietary supplements discusses effectiveness, FDA regulations, risks, quality, and talking to your doctor. Additionally, there are 16 CAM therapies summaries, a blog featuring cancer CAM topics and NCI’s Physician Data Query (PDQ®) summaries. There is a Spanish translation of the website.

Oncolink has information on all 7 selected CAM topics, but did not show on the first 2 Google results pages for any topic.

Oncolink states that the content is created and edited by its editorial staff composed of oncology nurses and social workers. Oncolink states that cancer experts at Penn Medicine assist with reviewing/writing the content, in addition to an editorial board (listed in the “About” section) that are responsible for reviewing the content. The publications in the CAM section provide an author, which on some pages is stated as “Oncolink team” and for others it’s the author’s name and credentials. Each webpage has a date when the information was last reviewed and evidence-based references. The site seems to be updated at least every 2 years.

Not Cancer Specific

Clinical Centers

Mayo Clinic

https://www.mayoclinic.org/drugs-supplements and https://www.mayoclinic.org/departments-centers/integrative-medicine-health/sections/overview/ovc-20464567

The DISCERN score was calculated to be 3.6.

The Mayo Clinic is a health system specializing in clinical care, research, and education. The website has a section on integrative medicine containing 10 summaries about: services, doctors, locations, travel/lodging, clinical trials, research, patient stories, costs/insurance, news, and resources. There are approximately 30 herb, supplement, and vitamin summaries containing an overview, evidence, Mayo Clinic’s opinion, safety, side effects, and interactions. Although this resource is not cancer specific, it does discuss integrative medicine for cancer. Some content is translated into a few languages, including Spanish.

The Google search showed Mayo Clinic on the first 2 pages for 1/4 CAM topics for which they have information.

Mayo clinic states that a team of medical experts create/edit the website. Individual authors are not provided. It is stated that the summaries are written by Mayo Clinic Staff, who are listed in the “About This Site” section with the names and credentials of the medical editors. Their production process has several steps: editorial research, editorial style, expert review, metadata annotation, and web production. Evidence-based references are provided at the end of each summary. Mayo clinic does not state how often updates occur, but a date is provided for the most recent update of each webpage, which appears to be updated regularly.

Government

National Center for Complementary and Integrative Health (NCCIH)

https://nccih.nih.gov/

The DISCERN score was calculated to be 4.2.

The Federal Government’s lead agency for scientific research on CAM, NCCIH provides information on integrative health products/interventions. The website has sections on health info, research, training, news/events, finding a practitioner, and practice guidelines. They have summaries for over 50 herbs and publications addressing specific health problems with background information, research/evidence, safety, side effects, and outside resources. Information is also provided in Spanish.

The Google searches did not show NCCIH on the first 2 pages of results for any of the 4 selected CAM topics for which they have information.

NCCIH states in the website information and policies section that the content is produced by NCCIH staff and reviewed by internal staff and external experts. Authors are not provided for each summary. Some publications acknowledge external experts (sometimes with credentials) that helped with reviewing. Details of the reviewing/updating process are not stated, but each page shows when it was last reviewed, which appears to be updated consistently. Evidence-based references are always provided.

NIH’s Office of Dietary Supplements

https://ods.od.nih.gov/

The DISCERN score was calculated to be 4.4.

ODS oversees various non-disease specific dietary supplement-related activities at NIH and disseminates information to the public. The website has 4 main sections: Health information, News & events, for researches, and about ODS. There is a list of approximately 180 supplements and alternative therapies with links to ODS factsheets and external information like NCCIH’s factsheets and NCI’s PDQ summaries. Most of the supplements/herbs have factsheets for patients and healthcare professionals. Some contain general summaries, while others have specific information, such as use with cancer therapy. There are sections for frequently asked questions, nutrient recommendations, and decisions-making. Information is also provided in Spanish.

The Google search did not show ODS on the first 2 results pages for any of the 4 selected CAM topics for which they have information.

This content is written, edited, and updated by staff within ODS that are listed in the “About ODS” section of the website. ODS states that health communication consultants, listed on the site, write the information, and update as needed. The dates and details of the editing history for each factsheet are provided. Authors and evidence-based references are not given for each factsheet.

Other

Natural Medicines

https://naturalmedicines.therapeuticresearch.com/

The DISCERN score was calculated to be 4.1.

Natural Medicines is a subscription-based database with dietary supplement and herbal therapy information. The database includes a quick search, interaction checker, effectiveness checker, and adverse effects search. After searching a topic, a list of monographs appears for professionals or patients in either English, Spanish, or French. These include background, use, safety, effectiveness, dosing, interactions, adverse effects, mechanisms of action, pharmacokinetics, and additional information. Other resources such as Susan G. Komen and Medline Plus use content from these monographs as educational material on their own sites.

Natural Medicines has information on all 7 selected CAM topics but did not show on the first 2 Google results pages for any topic.

Natural Medicines describes its team and editorial board as composed of expert editors, writers and contributors that are licensed health professionals and have clinical experience and training in evaluating research. The team is listed in the About Us section with names, job title, and credentials. Natural medicines states they use editorial principles and a literature review process to ensure high quality, evidence-based, and relevant content and performs daily updates. Every publication has evidence-based references and a date last reviewed/updated.

Discussion

Quality Assessment

Several tools have been developed to assess the quality of online health information resources.[15,16] The DISCERN tool was chosen for its ability to analyze information quality and trustworthiness and transparency of writing/vetting processes.

DISCERN was developed at the University of Oxford to evaluate the quality of consumer health information. The 15 questions assess the reliability and the quality of online healthcare treatment information. DISCERN focuses mostly on content and only has a few questions about the writing/vetting process. Some specific topics like transparency of authorship, editing, reviewing, and updating processes, were considered in each evaluation. Additional tools like Medline Plus’s tutorial on evaluating internet health information can be used to guide people searching for health information online.

Visibility

Google was selected as the main search engine during the initial search for resources that fit our criteria and for the final search to confirm the most visible websites. Previous research has shown that patients prefer to use general search engines like Google to search for health information, since they are more accessible and user friendly compared to tools like PubMed.

After the initial set of websites was chosen and analyzed, a final Google search was conducted to confirm the visibility of each website in the search results for several common CAM Cancer topics. The results obtained by the 5 searchers did not vary much. All searchers worked within OCCAM, and this may have contributed to the consistency of the results. The results and ranking of a Google search can vary based on the individual’s Google search algorithms and are dependent on several factors such as query meaning, relevance, content quality, usability of the websites, and other context and settings like location and search history (https://www.google.com/search/howsearchworks/algorithms/). Consistently throughout the 5 searchers’ results, websites like MSKCC, BreastCancer.org, Cancer Research UK and cancer.gov appeared the most on the first 2 results pages. Oncolink, NCCIH, ODS, and Natural Medicines were the least visible. NCCIH and ODS may not have appeared in these results because they are not cancer specific and the term cancer was used within each search, such as “acupuncture cancer.” Although these websites may not be in the top search results for these specific topics, they are useful online resources for CAM information.

The visibility and usage were also analyzed using the website traffic data of each resource. The data was obtained by contacting each website and requesting monthly or yearly traffic data for CAM specific webpages. Specific information for the traffic volume was often provided but considered confidential by the websites and thus cannot be published here. Some websites have their traffic data available publicly through the free version of tools like SimilarWeb and Alexa. These tools were not useful in this scenario however, as they did not provide specific data for the CAM-related pages. Based on the traffic data received from these resources, MSKCC, ODS, NCCIH, Breastcancer.org, Oncolink, and Cancer Research UK had the most pageviews/users for their CAM webpages, supporting them as highly consulted sites for CAM information. BCCT and CAM Cancer (NAFKAM) are less visible and viewed websites, but based on our analysis they have high quality, reliable CAM information and, given their editorial practices, are good models for other CAM online resources.

Gaps in resources

The online health information evaluations showed some aspects of certain websites that could be improved. Gaps seen in several of these resources were related to the writing and vetting process, such as lack of transparency with authorship and sources of information. Some resources did not provide enough information about who created, wrote, edited and reviewed each publication and the qualifications of those writers and editors. Some websites do not state the dates the publications were created, reviewed, or updated. Several of the resources also did not clearly state any references, and if they did provide a reference many did not have intext citations or hyperlinks to these sources. Table 3 gives a list of key features and questions to rapidly assess the quality of a CAM-information site. When the websites and publications do provide all this information, the reader can easily assess the reliability and quality of the resource and have greater confidence using such information in a dialog with their healthcare practitioners.

Table 3.

Key Characteristics for Online Health Information Resources.

Aims and relevance:	Why was this site created?
Sources of information:	References, website hyperlinks, in text citations
Writing, editing, reviewing process:	Transparency with process
Authors/editors qualifications:	Who created and updated the information and what are their credentials/qualifications?
Up to date publications:	Is the information up to date, is the last updated date provided?
Additional sources of support:	Are additional outside sources of support and information resources provided?
Shared decision making/complementarity:	Do they recommend consulting a healthcare professional and advice shared decision making?
Privacy and advertisement policies:	Privacy, advertisement, financial disclosure policies stated. What do they do with your information?

Limitations

Some limitations must be noted regarding our protocol and findings. We measured visibility 2 different ways, through Google result ranks and web traffic, to try to get an accurate understanding of the popularity of websites. Even with these measurements it is difficult to assess website popularity/visibility due to the complexity of the web traffic and the wide variety of factors that impact Google rankings. Another limitation is that the DISCERN analysis was conducted by one of the authors. Repetition of data collection and analysis by additional researchers would reduce any bias or score variations.

Future improvements

There is a need for unbiased, evidence-based, reliable, and informative material for cancer patients interested in CAM. Many people are unable to differentiate between reliable and unreliable sources, so it is important that CAM information websites frequently review and update their information and become more transparent with their information sources and writing processes. Health information websites can use resources like CAM Cancer (NAFCAM), BCCT, MSKCC, breastcancer.org, ODS, NCCIH and Cancer.gov as models to help attain higher quality and more trustworthy content. NCI’s Physician Data Query (PDQ®) is another trustworthy, evidence based online source of cancer information that is sponsored and maintained by NCI. It provides information targeted to health professionals and the general public on various cancer-related topics including integrative, alternative and complementary therapies (IACT). The PDQ’s IACT Editorial Board has produced and maintains information summaries on over 20 topics. The patient PDQ summaries contain commonly asked questions and general explanations of the therapy. The health professionals PDQ summaries contain reviews of the current, relevant, and reliable evidence for each topic. http://www.cancer.gov/publications/pdq.

Tools like DISCERN can be used as a guide for the standard principles of a trustworthy online health information resource. Patients and healthcare providers looking for reliable CAM resources can also use Medline Plus’s tutorial for evaluating internet health information to learn how to identify reliable and trustworthy online health information (https://medlineplus.gov/webeval/webeval.html).

Conclusion

Countless resources are available for people to use to educate themselves on aspects of CAM use for cancer patients. This can be very overwhelming and make it difficult to decipher between reliable and unreliable information, especially on the internet. It is vital that cancer patients and healthcare practitioners have trustworthy information sources to help inform their decisions and recommendations about the use of CAM therapies. It is our hope that this analysis can help guide those interested to reliable online resources to better educate the patients about the key factors one should look for when searching for health information online. Overall, the 11 websites analyzed had many trustworthy qualities with some aspects of the writing/vetting process and content that could be improved. The higher-quality, higher-scoring sources can be used as models for others and the guidelines followed in this analysis (DISCERN and Medline Plus) can be useful tools for individuals searching for health information online.