COVID-19 prevention and treatment information on the internet: a systematic analysis and quality assessment.

OBJECTIVE: To evaluate the quality of information regarding the prevention and treatment of COVID-19 available to the general public from all countries.
DESIGN: Systematic analysis using the 'Ensuring Quality Information for Patients' (EQIP) Tool (score 0-36), Journal of American Medical Association (JAMA) benchmark (score 0-4) and the DISCERN Tool (score 16-80) to analyse websites containing information targeted at the general public. DATA SOURCES: Twelve popular search terms, including 'Coronavirus', 'COVID-19 19', 'Wuhan virus', 'How to treat coronavirus' and 'COVID-19 19 Prevention' were identified by 'Google AdWords' and 'Google Trends'. Unique links from the first 10 pages for each search term were identified and evaluated on its quality of information. ELIGIBILITY CRITERIA FOR SELECTING STUDIES: All websites written in the English language, and provides information on prevention or treatment of COVID-19 intended for the general public were considered eligible. Any websites intended for professionals, or specific isolated populations, such as students from one particular school, were excluded, as well as websites with only video content, marketing content, daily caseload update or news dashboard pages with no health information.
RESULTS: Of the 1275 identified websites, 321 (25%) were eligible for analysis. The overall EQIP, JAMA and DISCERN scores were 17.8, 2.7 and 38.0, respectively. Websites originated from 34 countries, with the majority from the USA (55%). News Services (50%) and Government/Health Departments (27%) were the most common sources of information and their information quality varied significantly. Majority of websites discuss prevention alone despite popular search trends of COVID-19 treatment. Websites discussing both prevention and treatment (n=73, 23%) score significantly higher across all tools (p<0.001).
CONCLUSION: This comprehensive assessment of online COVID-19 information using EQIP, JAMA and DISCERN Tools indicate that most websites were inadequate. This necessitates improvements in online resources to facilitate public health measures during the pandemic. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Over 1200 websites were identified by using 12 search terms to improve the representativeness of returned COVID-19 websites.

A comprehensive analysis can be achieved using a combination of modified ’Ensuring Quality Information for Patients’ Tool, Journal of American Medical Association benchmark and the DISCERN Tool, all of which are respected and validated health information assessment tools.

Representativeness of search results of the Google search engine may be influenced by geographical factors and may differ when performed elsewhere.

Video-based health content was not within our scope and may provide a more comprehensive analysis of all online health information on COVID-19.

This study provides a snapshot of online health information as information on the internet is constantly changing.

Introduction

With the increasing popularity of the internet, both the accessibility and availability of health information grew drastically and have now become a primary source of information for many.1 2 It is known that health information-seeking behaviour also applies to the use of online resources and is ever more important during the current COVID-19 pandemic.3 Information on such a widely discussed topic will inevitably be vast and vary in production quality, potentially adversely affecting patient awareness and health-seeking behaviour.4 Many of these resources read by the public may be unreliable or produced from non-peer-reviewed sources and affect behaviours such as recognition of symptoms, taking appropriate preventative precautions or seeking timely treatment.3 5 6 Furthermore, inaccurate online information may contradict healthcare professionals and potentially compromise the trusting relationship with patients worsening outcomes.7

Since the declaration of COVID-19 as a pandemic on 12 March 2020, its prevalence and mortality have continued to rise8–10 and led to the introduction of various measures such as social distancing, quarantine procedures and lockdown protocols.11 As evidenced by previous outbreaks, effective public education and public health intervention rely on access to health information,12 13 which is now primarily delivered through the internet. Many countries have since introduced lockdown and quarantine protocols as their mainstay preventative measures14 but public health continues to be threatened by certain populations.15 Due to both the novelty and rapid developments of COVID-19, there is a significant barrier against the critical appraisal of online resources and, hence, necessitates a quantitative evaluation of the popular information sources available to the wider public.

Many instruments have been developed to evaluate patient information and may also be applied to online COVID-19 information.16 The modified Ensuring Quality of Information for Patient’s (EQIP) Tool is a reproducible modality used in previous studies to evaluate the reliability and quality of all information types, providing a robust assessment of quality, readability and design aspects of any written information.17–19 Previously, our group evaluated online information using the modified EQIP Tool in a variety of conditions and procedures including bariatric surgery,20 Dupuytren’s disease,21 carpal tunnel disease,22 breast augmentation,23 liposuction17 and liver transplantation.24 The Minervation validation instrument (LIDA, no acronym),25 Flesch Reading Ease Score and the Flesch-Kincaid Grade have also been used to evaluate the quality of online health information.26 However, they are not considered appropriate here as only readability and the website design are assessed, both of which are adequately covered by EQIP.27 Tools such as the Journal of American Medical Association (JAMA) benchmark and the DISCERN Tool (no acronym) have also been used to evaluate online health information and their combinational use can provide a more comprehensive evaluation.19 28 29 The internet has become an ever-important source of information and can determine health-seeking behaviour, which ultimately affects the progression of COVID-19. Hence, our study aims to assess the quality of information of top indexed websites that discuss information, prevention or treatment of COVID-19 using the modified EQIP Tool, JAMA benchmark and DISCERN Tool.

Methods

Eligibility criteria, information sources and data selection

On 27 March 2020, 12 search terms and phrases were queried on the most used search engine, Google,30–32 to obtain a database of websites. Only Google was used as previous studies have shown that the use of multiple search engines will only provide duplicate results. To increase the number of results, more search terms were used: ‘Coronavirus’, ‘COVID 19’, ‘Stop getting Coronavirus’, ‘Corona Virus’, ‘How to treat coronavirus’, ‘Coronavirus safety tips’, ‘Drugs for coronavirus’, ‘What is self isolation coronavirus’, ‘China virus’, ‘Wuhan virus’, ‘Coronavirus Medicine’ and ‘COVID 19 prevention’. These were commonly searched phrases identified using the ‘Google Adwords Keyword Planner’.33 Google AdWords allows the input of a term (in this case coronavirus), which then provides popular related keyword suggestions. The most popular search terms were ‘Coronavirus’, followed by ‘COVID’ and ‘Corona Virus’ and their respective search popularity peaked in mid-March and late-March 2020, respectively. Figure 1 summarises the most popular search trends. Only the first 10 pages of unique websites were identified and recorded as previous work suggests patients tend to stay within the first 100 returned webpages.18 24 Various search terms and their relative popularity were also collected directly from Google Trends34 for further comparative analysis.

Figure 1

Popular search terms used in COVID-19 and their relative popularity throughout the pandemic provided by Google Trends.

All websites written in the English language and providing information on prevention or treatment of COVID-19 intended for the general public or COVID-19 patients were considered eligible for inclusion. Any subsidiary pages or subdirectories of a website that contained information for the public and were easily accessible were also assessed. Websites or articles intended for professionals or specific population subsets, such as students alone, were excluded. Weblinks to purely video content, marketing content, daily caseload update or news dashboard pages with no educational purposes were excluded. The creation of the website database, eligibility assessment, website assessment and statistical analysis was performed between March and April 2020.

Website scraping

A website scraping tool was developed to identify and record all unique websites from the first 10 pages of Google results. The tool uses custom PHP: Hypertext Preprocessor (PHP) to make Hypertext Transfer Protocol (HTTP) requests to the search engine to mimic the requests made by the public. The queries were made from a server located in Texas, USA, but no preferences were made to limit searches by geographical region. The tool makes repeated requests, logs the first 10 pages of unique URLs and outputs the dataset after excluding all duplicate links within each search term. A minority of websites were restricted by General Data Protection Regulation and were accessed through the use of virtual private networks (VPN) as any websites that could reasonably be accessed by the general public were included.

Data entry

Six assessors (KSF, SAG, KHF, LL, AS and DR), all of whom fluent in English, independently assessed the websites between 30 March and 13 April 2020. The evaluation included 36 EQIP items and 4 items on JAMA benchmark, all assessed through ‘yes, no or N/A’ questions. DISCERN Tool adds a further 16 items to assess reliability and quality of treatment information using scales of 1–5. Assessors also recorded the country of origin, and type of source: Academic Centre, Charity/Non-Governmental Organisation, Encyclopaedia, Government/Health Department, Hospital, Industry, News Service, Patient Group, Practitioner and Professional Society. Organisations that primarily serve patients, such as Patient.info, is considered a ‘Patient Group’ whereas non-governmental organisations that oversee a broader demographic, like Red Cross and WHO, are classified as ‘Charity/Non-Governmental Organisation’. News service includes both primary and secondary news articles that are not written for professionals. ‘Practitioner’ considers the for-profit webpages of individual medical practitioners, whereas ‘Industry’ considers any for-profit organisation within the medical industry. ‘Academic Centres’ consider all sources from academic institutions, while ‘Professional Society’ refers to non-profit groups of healthcare professionals. Qualitative information about preventative methods and treatment was also recorded. After the initial round of data entry, each website was verified on a second round between 14 April and 21 April by a verifier with previous experience performing data entry for the evaluation of patient health information.

EQIP Tool

The EQIP Tool consists of 20 items, acting as a checklist for criteria such as quality of written work, design and coherence.35 More recently, modifications were made to the EQIP Tool, expanding the criteria to 36 items.36 This serves to satisfy both the guidelines of British Medical Association37 and International Patient Decision Aids Standards collaboration38 on the ideal patient information and the modified EQIP Tool have been used in a variety of specialities previously.17–19 A decision was made to use the modified EQIP Tool as the inclusion of ‘partly yes’ in the original EQIP introduces subjectivity into the responses and has been shown to lower its reliability.24 35 Thirty-six items across three domains were included: Content (items 1–18), Identification (items 19–24) and Structure (items 25–36). Similar to previous uses, ‘yes/no’ binary questions reduce assessor subjectivity in partial answers. ‘N/A’ option was also included if items were not relevant for the type of source. The Content domain assesses whether an adequate amount of information is included in an article, ranging from a description of the medical problem itself (items 1–3, 11 and 14) to the details of its management and complications (items 4–11). Identification domain assesses how well a website displays its production details, including date of issue, author, finance sources and bibliography (items 19–24). Structure domain evaluates the readability of a website and how well it accommodates its audience, such as delivering information through short, non-contradictory statements arranged in a logical layout (items 25–36). As COVID-19 is an emergent disease, certain items are tailored to accommodate for the limited evidence: describing treatment (item 3) include articles that address the lack of proven treatment, and alert signs (item 14) include recognised COVID-19 symptoms such as fever, cough and changes in taste or smell. A cut-off point of 75th percentile was set for EQIP Score to discriminate between high-scoring from low-scoring websites as was done in previous studies.

JAMA benchmark

Critical appraisals of internet resources are also assessed by one of the earliest core standards identified by JAMA in 1997.39 This checklist was proposed by Silberg et al to assist the appraisal and evaluation of the credibility of unregulated internet resources and have been used in various studies previously.40 41 This is evaluated by four items: Authorship, Attribution, Disclosure and Currency: Authorship requires identification of authors, their credentials and affiliations; Attribution requires appropriate citations on written information; Disclosure requires transparency of the website owner and conflicts of interests; Currency requires a clear indication of the date of publication and updates. Similarly, the ‘yes/no’ criterion is implemented to reduce the subjectivity of partial answers.

DISCERN Tool

The DISCERN evaluation tool was first developed in 1998 at Oxford to judge the quality of information regarding treatment choices.42 This tool has been validated and used across various specialities to assess treatment information.29 40 43 This consisted of 16 items to assess both the reliability and level of detail on treatments as well as the overall quality of the information. The reliability section (items 1–8) evaluates the ability of a website to achieve its aims while remaining unbiased and providing its sources of information. Quality section (items 9–15) evaluates the content specifically for describing the rationale, methods and alternatives to the current management of a disease. Criteria for treatment section was adjusted to accommodate the treatment uncertainty and items are considered fulfilled as long as the website discusses the relevant information with regards to potential drugs or interventions such as assisted ventilation. A score between 1 and 5 can be assigned to each item, with 1 being ‘no’, 3 being ‘partial’ and 5 being ‘yes’. To improve assessment accuracy, overall quality of information will be scored in proportion to the mean scores calculated from the answers to items 1–15, with 1 being the lowest and 5 being the highest.

Additional items

Additional items were included to further assess the websites, including whether a website discussed prevention methods, current treatments and the role of empirical evidence in the prevention or treatment for COVID-19. Data collected were in the form of ‘yes/no’ to reduce the ambiguity of partial answers. Additionally, details provided by the website on these items were recorded if the item scores ‘yes’. Websites were further analysed by whether their purpose is prevention, treatment, or both.

Statistical analysis

The dataset consisted of both continuous and categorical variables, which are reported as the mean, median and IQR as well as numbers and percentages, respectively. High-scoring websites are identified as those with scores above the 75th percentile for all three tools. Kruskal-Wallis tests were used for the analysis of continuous variables where appropriate and intraclass correlation coefficient (ICC) was calculated to identify the correlation between website analysis test methods. Fisher’s or χ tests were used to analyse proportions where appropriate. Inter-rater reliability of each assessor was evaluated using Bland-Altman plots. All p values were two tailed and considered significant when p<0.05. R V.3.3.2 (R Core Team, GNU GPL V.2 License), R Studio V.1.0.44 (RStudio, GNU Affero General Public License V.3, Boston, Massachusetts, 2016) and their respective graphical user interface rBiostatistics.com (rBiostatistics.com, London, Switzerland, 2017)44 was used to perform the statistical analysis.

Patient and public involvement

There was no patient or public involvement in the conception, design or data collection of the study or the production of the manuscript.

Results

Gathering of websites with information on COVID-19, its prevention and its management

A database of websites was gathered from the first 10 pages of unique URLs returned using the 12 search terms. The final dataset included 1275 URLs. After filtering out duplicate results and websites that failed to meet our inclusion criteria, 321 remained eligible for analysis. The workflow of dataset creation is shown in figure 2. The list of websites was obtained on a single day, 27 March 2020, and website evaluation was completed within 2 weeks.

Figure 2

Workflow of web scraping and exclusion: initial 1275 websites returned were filtered for duplicates and assessed for eligibility to include 321 websites in the final dataset.

Website demographics and search trends

COVID-19-specific searches regarding treatment and masks were significantly more popular (p<0.001) than prevention and peaked in mid-March and early April, respectively. The returned websites originated from 34 different countries (figures 3 and 4): the USA produced the most websites (n=178), followed by the UK (n=52), Australia (n=18) and Canada (n=18). However, no statistically significant differences were observed between the four countries across all tools. The source of information and website category is shown in table 1. News Services were the most common source of information (n=163), followed by Health Departments/Government (n=87).

Table 1

Website demographic and background information

Type	Total (n, %)	Treatment (n, %)	Prevention (n, %)	Prevention and treatment (n, %)
Academic Centre	10 (3.12%)	1 (0.31%)	4 (1.25%)	5 (1.56%)
Charity/Non-Governmental Organisation	12 (3.74%)	1 (0.31%)	9 (2.8%)	2 (0.62%)
Encyclopaedia	5 (1.56%)	0 (0%)	1 (0.31%)	4 (1.25%)
Government/Health Department	87 (27.10%)	1 (0.31%)	65 (20.25%)	21 (6.54%)
Hospital	7 (2.18%)	0 (0%)	5 (1.56%)	2 (0.62%)
Industry	30 (9.35%)	8 (2.49%)	10 (3.12%)	12 (3.74%)
Military	1 (0.31%)	0 (0%)	1 (0.31%)	0 (0%)
News Service	163 (50.78%)	34 (10.59%)	102 (31.78%)	27 (8.41%)
Patient Group	1 (0.10%)	1 (0.31%)	0 (0%)	0 (0%)
Professional Society	4 (1.25%)	0 (0%)	4 (1.25%)	0 (0%)
Research Centre	1 (0.31%)	0 (0%)	1 (0.31%)	0 (0%)
Total	321	46 (14.33%)	202 (62.93%)	73 (22.74%)

Country of origin of websites in descending order of the number of websites contributed.

Scores by top contributing countries (the USA, the UK, Canada and Australia) for EQIP Tool, JAMA benchmark and DISCERN Tool. EQIP, Ensuring Quality Information for Patients; JAMA, Journal of American Medical Association.

Forty-six websites (14.3%) described treatment methods alone, two hundred and two websites (62.9%) mentioned treatment methods alone and seventy-three websites (22.7%) discussed both. Of the prevention websites, 205 (63.9%) described social isolation, 169 (52.7%) physical distancing, 157 (48.9%) advised staying home and 136 (42.4%) described the benefits of disinfecting or cleaning surfaces. Of those that discussed treatment, 55 (17.1%) described the use of antiviral medications, 31 (9.7%) described hydroxychloroquine or chloroquine and 26 (8.1%) described the use of non-steroidal anti-inflammatory drugs such as paracetamol and ibuprofen. Only 31 (9.7%) websites discussed the use of oxygen, ventilation or fluids as a possible treatment method.

Overall performance

The mean total score for EQIP, JAMA and DISCERN are 17.78, 2.69 and 38.00, respectively, with a respective 75th percentile high-score cut-offs of 21, 4 and 43. No website achieved the maximum score for EQIP Content (out of 18) or Structure (out of 6) domain but one website did attain the maximum for EQIP Identification (out of 12). Seventy-four websites fulfilled all four JAMA criteria. Four websites achieved the maximum for DISCERN Reliability (out of 40) but none scored fully in DISCERN Treatment (out of 40). Seventy-four high-scoring websites were identified for EQIP and JAMA and seventy-six for DISCERN Tool. The mean scores of high-scoring and low-scoring websites for each tool and domain are as follows: EQIP Content (9.99 vs 6.07; p<0.001), EQIP Identification (4.03 vs 3.34; p<0.001), EQIP Structure (8.45 vs 6.96; p<0.001), Total EQIP (22.46 vs 16.37; p<0.001), Total JAMA (4.00 vs 2.30; p<0.001), DISCERN Reliability (31.72 vs 25.44; p<0.001), DISCERN Treatment (13.49 vs 10.31; p=0.002) and Total DISCERN (45.21 vs 35.76; p<0.001).

All tools and subsequent domains, except DISCERN Reliability and Total DISCERN scores, varied significantly between websites of different sources, notably with the Encyclopaedia (n=5) cohort holding the highest score across all domains. All tools and domains varied between website cohorts, with websites that discussed both Prevention and Treatment scoring above the mean values. Tables 2 and 3 summarise the variation of information quality with the source of information and website category, respectively. A detailed breakdown of the performance of each tool is displayed in online supplemental tables 1, 2 and 3.

Subset analysis of Ggovernment/Health Department and News Services

Collectively, 250 (77.9%) of all web links were either Government/Health Departments or News Services websites, of which 121 (37.7%) were based in the USA. Globally, there is significant variation between Government/Health Departments and News Services in EQIP Content (mean 8.11 vs 6.22; p<0.001), Total EQIP (mean 18.90 vs 17.06; p<0.001), Total JAMA (mean 2.16 vs 2.98; p<0.001) and DISCERN Treatment (mean 9.02 vs 11.72; p=0.001). Variations in US websites were similarly significant except for EQIP structure (p=0.148). The US-based cohort scored lower in Total EQIP than the global cohort but US News Services specifically scored higher in Total JAMA (3.17 vs 2.98) and Total DISCERN (39.25 vs 38.04). Breakdowns of comparison between the cohorts are provided in table 4.

Table 4

Comparison between Government and News Services

Indicator	Global (USA inclusive) (n=250)					USA (n=121)
	Government Mean Score	Government Mean Rank	News Mean Score	News Mean Rank	P value	Government Mean Score	Government Mean Rank	News Mean Score	News Mean Rank	P value
EQIP Content	8.11	161.34	6.22	106.37	<0.001	7.66	75.80	6.17	54.22	0.002
EQIP Identification	3.44	118.28	3.62	129.35	0.207	3.34	50.29	3.78	65.90	0.011
EQIP Structure	7.34	129.90	7.22	123.15	0.464	7.55	67.54	7.08	58.01	0.148
EQIP Total	18.90	151.52	17.06	111.61	<0.001	18.55	71.21	17.04	56.33	0.029
JAMA Total	2.16	85.72	2.98	146.73	<0.001	2.03	34.28	3.17	73.23	<0.001
DISCERN Reliability	27.47	137.15	26.31	119.28	0.062	26.53	60.26	26.66	61.34	0.876
DISCERN Treatment	9.02	110.95	11.72	133.27	0.001	8.84	47.99	12.59	66.96	0.001
DISCERN Total	36.49	105.32	38.04	136.27	0.963	35.37	54.72	39.25	63.87	0.182

EQIP, Ensuring Quality Information for Patients; JAMA, Journal of American Medical Association.

High-scoring websites

Sixteen websites scored above 75th percentile across all three evaluation tools, thirteen were from the USA, two from the UK and one from Canada. Most were from News Services (n=10), followed by industry (n=4), Encyclopaedia (n=1) and Government/Health Departments (n=1). The top five websites with the highest Total EQIP and Total DISCERN scores are shown in table 5 with their respective breakdowns. Top JAMA websites were not shown as there were 74 that scored the full 4 points.

Table 5

Top five websites based on EQIP and DISCERN scores

URL	Country	Treatment or Prevention	EQIP Total	JAMA Total	DISCERN Total
Top scoring EQIP sites
https://www.gov.je/health/coronavirus/Pages/index.aspx	Channel Islands (Jersey)	Both	29	3	49
https://en.wikipedia.org/wiki/2019%E2%80%9320_coronavirus_pandemic	USA	Both	28	3	61
https://www.health.nsw.gov.au/Infectious/alerts/Pages/coronavirus-faqs.aspx	Australia	Both	28	3	46
https://www.ageuk.org.uk/information-advice/coronavirus/coronavirus/%23	UK	Both	27	3	48
https://www.wikihow.com/Prevent-Coronavirus	USA	Prevention only	27	4	46
Top scoring DISCERN sites
https://www.sciencenews.org/article/coronavirus-covid19-repurposed-treatments-drugs	USA	Treatment only	22	4	75
https://www.vox.com/science-and-health/2020/3/4/21154590/coronavirus-vaccine-treatment-covid-19-drug-cure	USA	Both	22	4	74
https://en.wikipedia.org/wiki/Coronavirus_disease_2019	USA	Both	25	3	68
https://www.bloomberg.com/news/articles/2020-03-25/hydroxychloroquine-no-better-than-regular-covid-19-care-in-study	USA	Treatment only	21	3	65
https://www.theverge.com/2020/2/4/21122327/coronavirus-experimental-medication-treatment-wuhan-china-gilead-hiv	USA	Treatment only	18	4	64

EQIP, Ensuring Quality Information for Patients; JAMA, Journal of American Medical Association.

Intraclass correlation between tools

ICC between the three analysis tools is provided in online supplemental figure 1. The ICC between all three tools was moderate to high at 0.48 (95% CI 0.37 to 0.56). Furthermore, as the JAMA benchmark only offers four scoring variations, the Kruskal-Wallis analysis is used to correlate JAMA with EQIP and DISCERN. The results are shown in online supplemental figure 2 and demonstrate statistically significant moderate–high correlation for both JAMA–EQIP and JAMA–DISCERN.

Inter-rater reliability

The biases of each tool and assessor are within 95% CI limits. The Bland-Altman plots and individual degree of bias have been provided in online supplemental figure 3. The mean degree of bias and 95% CIs for each assessor is identified. Mean bias for EQIP, JAMA and DISCERN were −0.36,+0.29 and +0.51, respectively. Bias for each assessor, within each tool, was minimal and falls within their respective 95% CI.

Discussion

To our knowledge, this is the first study to evaluate the content intended for the public seeking information for preventing or treating of COVID-19. Our search terms included various synonyms of COVID-19 and ‘open’ search terms to capture the majority of materials related to our study. While no tools are validated to assess information specifically during pandemics; this study used a combination of EQIP, JAMA and DISCERN. The combined scope and efficacy of these tools enable a comprehensive evaluation of all the important aspects for a layperson seeking health information from articles; namely readability, coherence, design and quality of information.

Evaluation of the websites

We determined the quality of information to be low, as indicated by the low 75th percentile cut-offs for EQIP and DISCERN, at 21 and 43, respectively, or 58.3% and 53.8% of their respective full scores. Abundant COVID-19 content is being produced, as evident in figure 1, with the majority being excluded due to its nature rather than interterm duplications. Most frequently excluded websites were either caseload updates or general news articles lacking information on prevention and treatment.

Websites generally scored poorly, with an EQIP mean and median of 18 (IQR 15–20). Despite fewer marks allocated to Structure, websites generally outperform the Content domain (7.30 vs 6.97), suggesting information quality is less adequate than usability. The indicators assessing referencing quality, such as JAMA benchmark and DISCERN Reliability, scored similarly at 2.69 (67.3% of maximum) and 26.93 (67.3% of maximum), respectively, but scored lower in the EQIP Identification domain 3.50 (58.3% of maximum), likely due to more items (ie, whether patients were involved in the material’s production). Generally, high-scoring websites performed better in the Content domain, with OR between 3 and 7. The OR was significantly higher for item 4 (defining the purpose of interventions) (OR 27.78; 95% CI 4.695 to 1000; p<0.001), suggesting that high-scoring websites provided greater reasoning behind preventative and treatment measures. While high-scoring websites scored significantly higher across 23 of the 36 EQIP items, quantitative benefits (item 8) is a notable exception where both cohorts performed poorly (4.1% vs 2.4%; OR 1.695; 95% CI 0.267 to 8.197; p=0.436) and was significantly lower than the scores in available literature.18 23 24 This likely reflects the general lack of COVID-19 knowledge compared with previously explored diseases and treatments. High-scoring websites similarly performed better in Identification (OR ranged between 1.312 and 5.376), with the inclusion of bibliography (item 23) differing most (41.89% vs 11.74%; OR 5.376; 95% CI 1.727 to 7.407; p<0.001) as the majority of websites lacked bibliographies, potentially due to subpar production quality in a high turnover topic. Structure revealed both high-scoring and low-scoring websites to provide clear information (item 30; 98.65% vs 89.88%; OR 8.197; 95% CI 1.294 to 333.3; p=0.013). High-scoring websites provided poor benefit−risk balancing (item 31; 39.19% vs 10.93%; OR 5.208; 95% CI 2.703 to 10.101; p<0.001), as did most other websites assessed (17.5% overall), comparable to existing studies ranging from 11% to 44%. As shown in figure 5, EQIP scores were relatively homogeneous, ranging from 6 to 29, with the majority between 14 and 22. Overall performance agreed with available literature that online health information is inadequate, as median EQIP scores ranged between 15 and 19 and IQR ranged 12–20 and 16–22.18 20 22 23 45 Furthermore, scores for describing intervention sequence (item 6) and quantitative risks (item 10) were much lower (15.9% and 0.9%, respectively) when compared against the study on gallstone disease (27% and 21%, respectively) or liver transplantation (66% and 53%, respectively). This likely reflects the prioritisation of discussing prevention and treatment method efficacy over treatment sequences and risks. Item 23 also scored poorly, both in itself and against literature: only 18.7% of websites provided a short bibliography whereas studies ranged from 19% in liver transplantation to 47% in orthognathic surgery. Interestingly, COVID-19 websites scored well in dating (item 19; 87.2%) compared literature, likely reflecting the demanding and time-sensitive nature to understand how the COVID-19 pandemic affects individuals. Unsurprisingly, encyclopaedias (n=5) scored the highest in content (10.80), identification (4.80) domains and overall EQIP (24.20), holding the highest proportion of websites discussing both prevention and treatment (n=4; 80%). Encyclopaedias’ high scores are attained through their endeavours to provide neutral summaries which meet the majority of the EQIP criterion, as well as the items in both JAMA benchark and DISCERN tool.46 Nonetheless, while overall EQIP scores do not differ substantially from the existing literature, it is important to recognise that the quality of information is significantly influenced by the rapid turnover of information, a phenomenon not present in any of the previous studies.

Figure 5

High-score distribution of the final dataset compared against low-scoring websites for EQIP Tool, JAMA benchmark and DISCERN Tool. EQIP, Ensuring Quality Information for Patients; JAMA, Journal of American Medical Association.

JAMA benchmark scores, unlike EQIP and DISCERN, were more consistent as only four points are available. Hence, high-scoring websites often fulfilled all four JAMA criteria, whereas the majority of other entries scored two or three. Low-scoring websites scored significantly lower (p<0.001) in Authorship and Attribution, with only 44.13% and 17.81% fulfilling the criteria, reflecting congruence with the findings from EQIP regarding bibliography inclusion. Contrarily, website ownership and funding assessment, under Disclosure (86.23%), and assessments of publishing and updating dates, under Currency (81.78%), varied to a lesser degree, although still scoring significantly lower (p<0.001). Comparative to the related EQIP sections, Disclosure and Currency likely scored better by assessing attributes independent to the content itself. Professional societies (n=4; 3.50), closely followed by encyclopaedias (n=5; 3.40), scored the highest overall, whereas the mean JAMA score was only 2.69, with 4 websites scoring 0 points. This is likely attributable to the lack of additional assessment criteria by the JAMA benchmark as it was developed during the infancy of web-based resources. Regardless, the tool is still effective at identifying high-quality content as high-scoring websites scored significantly better across each item (p<0.001).

DISCERN’s mean score of 38.00 across 16 items averaged 2.38 out of 5 per item. The DISCERN handbook details a rating of one when the information does not provide the appropriate information, three where it addresses it partially and five for a complete and adequate inclusion. Based on these guidelines, the majority of websites meet the listed criteria to a minimal extent. Similar to EQIP, DISCERN scores vary significantly, ranging from 19 to 75, with the majority distributed between 25 and 47. Of the 16 items, 12 presented statistically significant differences between high-scoring and low-scoring websites. Interestingly, the Reliability section provided the greatest difference in scoring, seen between item 4, clear display of information sources (mean 4.16 vs 2.55; p=0.019), and item 5, a clear indication of where sources were used (mean 3.54 vs 2.28; p=0.007). This corroborates with results from EQIP and JAMA assessments in highlighting the inadequacies of informative material production.

Quality of treatment information section scores varied less, although all with statistically significant differences. The greatest differences were in item 9, describing how each treatment works (1.76 vs 1.43; p=0.005) and item 10, detailing treatment benefits (1.78 vs 1.43; p<0.001). Despite accommodating the treatment uncertainty during the assessment, almost all items evaluating treatment scored poorly, including high-scoring websites, and may be reflective of its lower journalistic priority or demand as the majority of websites were tailored towards discussing prevention. As a whole, DISCERN scores specifically addressing treatment options were poor, scoring 45.07 (SD 11.68) for websites that only discuss treatment as well as 42.84 (SD 10.93) for websites that discuss both prevention and treatment. These scores are also lower than those of other DISCERN studies, which have mean scores between 45.8 and 56.1, with SD between 8.76 and 13.6.47–49 While the majority of DISCERN scores were low, websites scored similarly in the Reliability section. The mean scores for items 1, 2 and 3 were the highest of all indicators, averaging 4.37, 4.29 and 4.33, whereas the items in treatment section scored between 1 and 2, with a maximum mean score of 1.66 in Item 14. The treatment section of DISCERN shows much more variation, with prevention alone (8.60; n=202) scoring lowest, treatment alone (18.09; n=46) scoring highest and websites discussing both (13.47; n=73) in between. This suggests that many websites do not include treatment information and, of those that do, websites tend to avoid discussion rather than provide the limited information available.

In short, all three tools used are validated based on international recommendations and provide a comprehensive assessment of online information: EQIP delivers an all-rounded evaluation of health information, DISCERN excels at scrutinising treatment and JAMA benchmark assesses all of a website’s content as a whole. Additionally, as design alone is known to improve the perception of information credibility,50 the higher emphasis on quality of content in EQIP and DISCERN will likely prevent well-presented and user-friendly websites with poor content from attaining high scores over poorly presented websites with good content. Altogether the tools suggest that the majority of COVID-19 websites are generally of poor quality and that quality fluctuates highly. This may subsequently impair the ability of the public to filter out websites of low validity and reliability and, hence, increase their risk of unsafe health behaviours during the pandemic. Through the use of these tools, the same standards can be held across different sources of information to produce higher quality educational material to improve both education and awareness.

Reasons for poor quality information

Most sources scored poorly due to several possible reasons. First, as COVID-19 was only identified in early December 2019,51 and recognised as a pandemic in March 2020,52 the general lack of information to produce patient information prevents accurate and reliable conclusions to be drawn. While research efforts have since gained traction, production of research, reviewing and publication is a lengthy process comparative to the rapid spread of COVID-19. Journals have since implemented fast tracking of COVID-19 research.53–55 Similarly, Governments and Health Departments have also cooperated to support and fast-track COVID-19 studies.56 The combined efforts have facilitated the publication of over 5000 COVID-19 articles in the WHO database alone.57 It is also important to note the role of preprint servers during the pandemic as they are accessible to the public. While improving accessibility helps facilitate peer reviewing; non-peer-reviewed articles can potentially be used, or cherry picked, by non-professionals that can adversely affect public understanding.58–61

Health literacy also plays a crucial role in how COVID-19 information influences health behaviour. The associations between health literacy and health behaviour are well documented: low literacy is common among older adults engaging in poor health behaviour (p<0.005) ranging from lifestyles, such as physical activity, dietary habits and obesity, to social factors, such as loneliness and social isolation.62–64 The effects of primary preventative measures, such as social distancing, self-isolation and other hygiene recommendations, may become impaired by the abundance of poorly written and incorrect information online. In particular, the elderly, most vulnerable population, suffer from even higher risks due to their lower health literacy.65 Although health literacy, and by extension, health behaviours, can be improved through education,66 effective dissemination of credible information is critical during the pandemic. The public, and journalists, need to exercise caution when accessing research and preprints during this period as inadequate health literacy may lead to counterproductive effects. Similarly, mass production of online information greatly increases the difficulty in distinguishing reliable information from the sea of misinformation, and hence, a greater emphasis must be placed on authors and journalists to deliver unbiased, credible and accurate information to the public.67 68

On publication, articles are reviewed and summarised by journalists who bridge the knowledge gap between scientists and the wider public. This allows efficient dissemination of critical research to those who lack the scientific background to critically appraise and evaluate research. Notably, while 52% (n=169) of websites were ‘News Services’, only 7.1% (12) were considered high scoring, reflecting that very few provide a comprehensive account of COVID-19 information, possibly explained by the difficulty in matching the pace of COVID-19 research and technical inadequacies in delivering accurate and concise scientific information. As health information-seeking behaviour of the public will likely be based on news services,69 the highly variable and generally poor content is problematic.

Previous studies have identified that many health journalists lack the training required to accurately disseminate health news, leading to potentially harmful health effects.70 71 The low scores across all tools indicate overall inadequacy of both reliability and accuracy. A survey of medical journalists across 37 countries highlighted the 3 most common barriers against quality content: lack of time, space and knowledge.72 While journalism stresses the ability to summarise content quickly and concisely, experienced journalists often report the lack of knowledge as a barrier. The lack of expert knowledge in a rapidly progressing scientific field can impact the quality of conveyed information significantly. Furthermore, journalists reported difficulty in finding experts to explain the jargon, further impacting quality. This presents a large barrier towards disseminating quality COVID-19 information as the redirected efforts of many countries and institutions’ scientists into research reduces availability to assist with medical journalism.73 74 Expertise in a relevant academic background likely helps improve the content, as indicated by the highest-scoring entry, whose author holds a PhD in molecular genetics which scored 22 in EQIP, 4 in JAMA and 74 in DISCERN.75 In short, our findings highlight the importance of addressing health outcomes through health literacy of both the public and health journalists.

Our analysis identified the majority of websites as sources from Governments or health departments across various states of the USA, many of which have based information and advice from the Centers for Disease Control and Prevention (CDC). As both the nature and purpose of these sources vary, the visiting population and demographics would also vary due to differences in information-seeking behaviours. A recent study has identified a deficit of awareness and trust in information originating from the Government such as those of the CDC, revealing that while up to 83.6% of American adults are aware of the CDC, only 64.6% trust this source.76 American adolescents were, however, less aware of the CDC (55.8%), but were more trusting of their information (72.2%). As previously established, trust in a Government is predictive of health outcomes as it affects behaviours such as service usage and vaccination rates.77–79 Similarly, as access and usage of online health information vary between different demographics, it is paramount that we create and provide targeted and effective educational material for public use.80 81

A comparison between the global Government/Health Department and News Services websites revealed significant differences between EQIP Content, Total EQIP, Total and DISCERN Treatment. The EQIP Content scores reflect differences in where information is obtained, with Government/Health Departments using primary research, whereas journalists tend to use secondary research. However, News Services scored higher in DISCERN Treatment, potentially due to the Government’s reluctance to prematurely disclose treatment information at early stages of discovery, whereas journalists may freely report results of all potential studies. Interestingly, the majority of US Government websites had some form of copy-pasted information from the CDC, likely with the aim of maintaining consistency and centralisation information sources. In contrast, the majority of other sources rewrite information based on a variety of sources. American sources display a similar pattern of variation but the EQIP Identification scores of Government/Health Departments show statistically significant differences and scores lower than News Services (mean 3.34 vs 3.78; p=0.011). The verbatim use of CDC information on these Government websites may have neglected the importance of clarity, transparency and critical appraisal of literature and focused on disseminating information instead.

Is the current information online adequate?

Google Trends identified the search popularity of treatment to have increased rapidly, outgrowing prevention searches since early March. This suggests that initial demands for preventative information have been sufficiently met and interests now shift towards treatment. The 275 websites addressing prevention scored below the mean EQIP (17.12 vs 17.78; p<0.001), JAMA (2.53 vs 2.69; p=0.001) and DISCERN scores (34.64 vs 38.00; p<0.001), showing that quality of preventative information remains subpar across all indicators. Similarly, treatment websites demonstrated their efficacy in sourcing, over discussing information, as they excelled in Identification (3.87 vs 3.50; p<0.001) but scored poorly and below the mean in EQIP (16.89 vs 17.78; p<0.001). This is also reflected by its above-average JAMA (3.13 vs 2.69; p=0.001) and DISCERN (45.07 vs 38.00; p<0.001) scores. Contrarily, the 73 websites that discussed both prevention and treatment of COVID-19 consistently scored higher than the mean across all indicators: EQIP (20.15 vs 17.78; p<0.001), JAMA (2.85 vs 2.69; p=0.001) and DISCERN (42.84 vs 38.00; p<0.001). These websites explored multiple aspects of the virus and likely use a variety of sources, thus producing higher quality articles through a better understanding of the topic. To combat increasingly dangerous COVID-19 myths, such as injection of disinfectants as treatment, continued maintenance and improvement to online available resources is paramount.82 83

Limitations

Although the ‘Google’ search engine is the most commonly used, it is not wholly representative as searches are often affected by the location of the requesting server and previous internet usage. While querying from every country is not practical, the server used in the study was not used to conduct any other COVID-19-related searches, thus impact should be minimised. A further search was conducted on 10 July 2020 on the top 10 scoring EQIP and DISCERN websites through VPN servers in Texas, London, Toronto and Sydney. Only three of the websites were found within the first 10 pages of results using the same search terms on the original server, with London and Sydney each returning two of the results, Toronto returning none (online supplemental table 4) and a remaining URL redirecting to another page. The search also confirms differences in results between the location of the search; however, all three results from the Texas server were covered by London and Sydney. As websites can be updated or removed any time, our results are representative only at the time of the search, demonstrated by the distinct lack of the original websites in our second search in July. Similarly, search terms obtained using ‘Google Adwords Keyword Planner’ and ‘Google Trends’ may not truly be indicative of search patterns of the wider public. Hence, 12 different variations of ‘COVID-19’ names and phrases potentially used by patients were used as search terms. While social media is increasingly used to share health information, our study focused on search engines because previous studies have identified a lack of trust in social media information and which acts as a barrier against public engagement.84 Similarly, while video-based information constitutes a valuable source of information, to the best of our knowledge, no tools have been validated in assessing video-based information, particularly for COVID-19. Forced inclusion of video content would likely yield inaccurately low scores across current tools as videos typically do not include as much written information. Another limitation is the exclusion of non-English language websites, especially reducing the representation of publicly available information given the international nature of the pandemic. A minority of URLs, particularly of News Services, regularly update their content or redirect visitors, potentially affecting the second round of evaluation and subsequent statistical analysis. Additionally, modified EQIP Tool, JAMA benchmark and DISCERN Tool were not designed to specifically assess the highly variable information produced during pandemics. However, the EQIP Tool was designed to assess any type of patient information and demonstrated high inter-rater reliability.17 Similarly, JAMA benchmark was designed to evaluate website reliability alone and DISCERN examines both reliability and content accuracy, thus, the combinational use of tools enhances the accuracy and objective assessment of these websites.

Conclusion

In short, the abundance of internet resources providing COVID-19 information is exemplified by the numerous identified websites during our search. The information available to the public may affect their health decisions, which, subsequently, affects the efficacy and outcome of public health measures implemented by health departments. As effective treatments and vaccine research is underway, COVID-19 is primarily addressed with preventative measures, hence necessitating a critical review of the quality and nature of the information accessible to the public. Our results demonstrated that the websites were chiefly produced by News Services and Government/Health Departments but were nonetheless of low quality. While the majority of websites addressed prevention, and likely met the information needs of the public as reflected by search trends, there is a relative deficit in websites that discuss treatment methods. A minority of websites discussed both prevention methods and treatment and were generally good resources but the majority of websites were also of inadequate quality. Thus, there is a need for higher quality online COVID-19 resources to facilitate public education and enable better cooperation and outcomes of public health measures.

Table 2

Overall quality of information of all sources of information

Indicator/ (mean, SD)	Academic centre	Charity/Non-Governmental Organisation	Encyclopaedia	Government/Health Department	Hospital	Industry	Military	News Service	Patient group	Professional society	Research Centre	P value
EQIP Content	7.70 (2.50)	6.75 (3.33)	10.80 (3.11)	8.11 (2.49)	7.14 (2.19)	7.13 (2.60)	5.00 (0)	6.22 (2.11)	10.00 (0)	5.50 (3)	5.00 (0)	<0.001
EQIP Identification	2.90 (1.20)	3.08 (0.90)	4.80 (0.45)	3.44 (1.03)	2.71 (1.38)	3.4 (1.13)	1.00 (0)	3.62 (0.73)	3.00 (0)	4.00 (0)	4.00 (0)	0.003
EQIP Structure	7.90 (1.85)	7.67 (1.30)	8.60 (1.67)	7.34 (1.45)	7.43 (0.98)	6.97 (1.83)	9.00 (0)	7.22 (1.39)	10.00 (0)	7.25 (0.5)	5.00 (0)	0.08
EQIP Total	18.50 (3.66)	17.50 (4.32)	24.20 (3.56)	18.9 (3.79)	17.29 (3.50)	17.5 (4.48)	15.00 (0)	17.06 (3.19)	23.00 (0)	16.75 (3.4)	14.00 (0)	<0.001
JAMA Total	2.50 (0.97)	2.50 (1.00)	3.40 (0.55)	2.16 (0.86)	2.14 (1.57)	2.73 (1.23)	1.00 (0)	2.98 (0.85)	3.00 (0)	3.50 (0.58)	3.00 (0)	<0.001
DISCERN Reliability	27.20 (6.78)	26.25 (6.51)	35.60 (4.77)	27.47 (5.96)	29.86 (5.58)	26.6 (7.46)	23.00 (0)	26.31 (6.3)	27.00 (0)	29.50 (4.2)	26.00 (0)	0.128
DISCERN Treatment	13.00 (6.70)	9.42 (2.39)	16.00 (9.30)	9.02 (3.14)	8.71 (1.89)	13.33 (6.39)	8.00 (0)	11.72 (6.22)	13.00 (0)	8.50 (1)	17.00 (0)	0.001
DISCERN Total	40.20 (10.38)	35.67 (7.67)	51.60 (12.3)	36.49 (7.50)	38.57 (5.47)	39.93 (10.45)	31.00 (0)	38.04 (10.44)	40.00 (0)	38.00 (5.1)	43.00 (0)	0.167

EQIP, Ensuring Quality Information for Patients; JAMA, Journal of American Medical Association.

Table 3

Overall quality of information of all websites subsets

Indicator (mean, SD)	Overall	Treatment	Prevention	Prevention and Treatment	P value
EQIP Content	6.97 (2.52)	6.26 (1.81)	6.55 (2.30)	8.58 (2.84)	<0.001
EQIP Identification	3.50 (0.93)	3.87 (0.72)	3.32 (0.97)	3.78 (0.80)	<0.001
EQIP Structure	7.30 (1.47)	6.76 (1.46)	7.25 (1.46)	7.79 (1.37)	<0.001
EQIP Total	17.78 (3.71)	16.89 (2.84)	17.12 (3.45)	20.15 (3.95)	<0.001
JAMA Total	2.69 (0.98)	3.13 (0.72)	2.53 (1.05)	2.85 (0.84)	0.001
DISCERN Reliability	26.93 (6.35)	26.98 (6.92)	26.04 (5.72)	29.37 (7.04)	<0.001
DISCERN Treatment	11.07 (5.60)	18.09 (6.04)	8.60 (2.45)	13.47 (6.73)	<0.001
DISCERN Total	38.00 (9.61)	45.07 (11.67)	34.64 (6.52)	42.84 (10.93)	<0.001

EQIP, Ensuring Quality Information for Patients; JAMA, Journal of American Medical Association.