YouTube as a source of information on COVID-19: a pandemic of misinformation?

INTRODUCTION: The COVID-19 pandemic is this century's largest public health emergency and its successful management relies on the effective dissemination of factual information. As a social media platform with billions of daily views, YouTube has tremendous potential to both support and hinder public health efforts. However, the usefulness and accuracy of most viewed YouTube videos on COVID-19 have not been investigated.
METHODS: A YouTube search was performed on 21 March 2020 using keywords 'coronavirus' and 'COVID-19', and the top 75 viewed videos from each search were analysed. Videos that were duplicates, non-English, non-audio and non-visual, exceeding 1 hour in duration, live and unrelated to COVID-19 were excluded. Two reviewers coded the source, content and characteristics of included videos. The primary outcome was usability and reliability of videos, analysed using the novel COVID-19 Specific Score (CSS), modified DISCERN (mDISCERN) and modified JAMA (mJAMA) scores.
RESULTS: Of 150 videos screened, 69 (46%) were included, totalling 257 804 146 views. Nineteen (27.5%) videos contained non-factual information, totalling 62 042 609 views. Government and professional videos contained only factual information and had higher CSS than consumer videos (mean difference (MD) 2.21, 95% CI 0.10 to 4.32, p=0.037); mDISCERN scores than consumer videos (MD 2.46, 95% CI 0.50 to 4.42, p=0.008), internet news videos (MD 2.20, 95% CI 0.19 to 4.21, p=0.027) and entertainment news videos (MD 2.57, 95% CI 0.66 to 4.49, p=0.004); and mJAMA scores than entertainment news videos (MD 1.21, 95% CI 0.07 to 2.36, p=0.033) and consumer videos (MD 1.27, 95% CI 0.10 to 2.44, p=0.028). However, they only accounted for 11% of videos and 10% of views.
CONCLUSION: Over one-quarter of the most viewed YouTube videos on COVID-19 contained misleading information, reaching millions of viewers worldwide. As the current COVID-19 pandemic worsens, public health agencies must better use YouTube to deliver timely and accurate information and to minimise the spread of misinformation. This may play a significant role in successfully managing the COVID-19 pandemic. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

YouTube has been a significant source of misleading information during public health crises, including the H1N1, Ebola and Zika outbreaks.

These studies reported that approximately 23%–26% of YouTube videos were misleading, while videos from reputable sources were vastly under-represented.

More than one-quarter of the most viewed YouTube videos contained misleading information, eliciting over 62 million views worldwide.

Videos from reputable sources remain under-represented in the current COVID-19 pandemic.

Strategies that can be employed by government and public health agencies to increase the viewership of their quality content on COVID-19 were identified.

Misinformation about COVID-19 is reaching more individuals than in past public health crises, as YouTube continues to grow as a source of health information.

As the current COVID-19 pandemic worsens, public health agencies must better use YouTube to deliver quality content and minimise the spread of misinformation.

Introduction

Coronavirus disease 2019 (COVID-19), first detected late December 2019,1 2 is the first coronavirus (CoV) pandemic of the 21st century. It is the most serious CoV outbreak in history with 266 073 cases and 11 183 deaths as of 21 March 2020,3 outnumbering both the 2003 severe acute respiratory syndrome epidemic (8096 cases, 774 deaths) and the 2012 Middle East respiratory syndrome outbreak (2494 cases, 858 deaths).4 5

Over 70% of adults search the internet for health and healthcare related information.6 YouTube is one of the most dominant sources of online information, with over 2 billion users generating billions of daily views.7 The strength of YouTube, as compared with other social media platforms, lies in its judicious use of audio and visual communication, making it readily accessible to individuals from all demographic backgrounds. Although YouTube is a powerful educational tool that healthcare professionals can mobilise to disseminate information and influence public behaviour, if used inappropriately, it can simultaneously be a source of misleading information that can work significantly against these efforts.

Previous studies have shown that YouTube has been a source of useful and misleading information during public health crises, including the H1N1 pandemic, Ebola outbreak and Zika outbreak.8–10 These studies found that approximately 23%–26.3% of YouTube videos informing viewers about these diseases were misleading and that these videos largely originated from independent users.8–10 Furthermore, they demonstrated that reputable videos were under-represented.8–10 Although these studies provide preliminary insights into the use of YouTube during public health crises, the generalisability and applicability of their findings to the current COVID-19 pandemic are limited. First, some studies8 11 did not use validated instruments to assess the reliability and quality of videos. In addition, given the rapid evolution and increasing usage of social media, public viewership habits and public health officials’ utilisation of YouTube may be substantially different during the current COVID-19 pandemic, as compared with the time of publication of these studies. Currently, the literature lacks evidence on YouTube as a source of information in the context of the COVID-19 pandemic. Insights into the quality of information during the COVID-19 pandemic is particularly essential, given that the current scale of COVID-19 has greatly surpassed both the H1N1 pandemic and past CoV outbreaks.3–5 12

The primary objective of this study is to evaluate the accuracy, usability and quality of the most widely viewed YouTube videos on COVID-19. Based on these results, the second objective is to propose immediate recommendations that will enable professional organisations to better utilise YouTube and expand the delivery of accurate and timely information, as the COVID-19 pandemic evolves.

Methods

Search protocol

The following search method was developed a priori and has been validated in previous studies of YouTube content.8–11 13 14 A YouTube search was performed on 21 March 2020 using the keywords ‘coronavirus’ and ‘COVID-19’. The search was done using a cleared-cache web browser, which consists of the most recent version of Google Chrome in incognito mode with all available updates installed. Search results were sorted by ‘views’ to obtain the most widely viewed YouTube videos in descending order at the time of search. The hyperlinks of most widely viewed 75 videos for each keyword were collected on a spreadsheet for data collection.

Study selection and data extraction

Two reviewers (HO-YL and AB) independently reviewed and analysed all 150 videos. The exclusion criteria included duplicate videos, non-English videos, videos without audio and/or visual information, videos exceeding 1 hour in duration, livestream videos and videos unrelated to COVID-19. This exclusion criteria were based on previous studies9 10 13 with the addition of livestream videos, as we felt that they are not representative of the majority of YouTube videos represented in our sample. Only English videos were included, as English is the universal language adopted by many countries globally.15 Any discrepancies between reviewers were resolved by discussion with the third reviewer (DH) and senior author (JC) for consensus. Descriptive characteristics of all videos including video title, video hyperlink, number of views, number of likes and dislikes, upload date of video, video duration and video publishing category were included. Video publishing categories were adapted from previous work of YouTube content (online supplementary table S1).11

The reliability and quality of video content was assessed using the modified DISCERN score (mDISCERN) (online supplementary table S2) and the modified Journal of the American Medical Association (mJAMA) benchmark criteria (online supplementary table S3).16 17 There were not any validated evaluation tools available to assess online information regarding COVID-19. Thus, to assess the usefulness of video content for the average viewer, we created a novel five-point COVID-19 Specific Score (CSS). One point was awarded if the video contained exclusively factual information on the transmission, typical symptoms, prevention strategies, potential treatments and epidemiology of COVID-19 as of the video’s publication date, for a total of 5 points (online supplementary table S4). This score was developed based on previously published work on public health emergencies,8–11 a review of the COVID-19 literature and expert input.

Patient and public involvement

No patients and public were involved in any way in this study.

Data analysis

Following data extraction, two independent reviewers (HO-YL and AB) categorised all videos into factual and non-factual videos and into their video publishing category in duplicate, with third reviewer (DH) and senior author (JC) consultation for consensus. Videos were considered non-factual if they contained one or more non-factual statements, as evaluated based on guidelines from major public health agencies (eg, WHO and Centers for Disease Control and Prevention) at the video’s publication date. Subsequently, the CSS, mDISCERN and mJAMA scores of each individual video were calculated. Interobserver agreement was evaluated with the Cohen’s kappa coefficient. Two tailed t-tests were used to compare factual videos to non-factual videos, while a one-way analysis of variance and Dunnett’s post hoc test were used to test for differences between video sources in comparison with videos from government and professional sources (control group), with a significance level of p=0.05. Government and professional videos were combined in group analysis, as these videos sources were considered to have the highest standard of publication. All statistical analyses were performed using SPSS software (SPSS V.19.0).

Results

Video characteristics

Of the 150 videos screened and 69 videos (46%) were identified for inclusion. The remaining 81 videos (54%) were excluded for duplicates (42), non-English language (31), over 1 hour long (4), live streams (1), no audio (2) or unrelated to COVID-19 (1), as shown in figure 1. The total viewership of the 69 included videos was 257 804 146 views.

Figure 1

Flow diagram for review of YouTube videos on COVID-19.

Source of videos

The 69 videos were categorised into eight categories as shown in figure 2. Cohen’s kappa for the categorisation of videos into their respective categories was 0.90±0.04. The total viewership of our videos was 257 804 146 views with 29% from network news, 22% from consumers, 21% from entertainment news, 12% from internet news, 7% from professionals, 5% from newspapers, 2% from education and 2% from government sources.

Figure 2

Video publishing source of included videos.

Factual versus non-factual videos

Of the 69 included videos, 19 (27.5%) contained non-factual information, while 50 (72.5%) contained only factual information. Cohen’s kappa for the categorisation of factual and misleading videos was 0.86±0.07. Non-factual videos accounted for 24.1% of the total viewership for a total of 62 042 609 views. Factual videos accounted for 75.9% of total viewership.

When comparing both groups, there were not any significant differences in views per video (p=0.497), likes per video (p=0.785), dislikes per video (p=0.104), duration (p=0.462) and days since publication (p=0.230) as shown in table 1.

Table 1

Comparison of factual and non-factual videos on COVID-19.

Category	Non-factual	Factual	P value
# of videos	19 (27.5)	50 (72.5)
# of views	62 042 609 (24.1)	195 761 537 (75.9)
# views/video	3 265 400±2 093 369	3 915 231±3 925 941	0.497
# of likes/video	68 003±77 411	76 292±1 22 808	0.785
# of dislikes/video	4122±4121	2702±2780	0.104
Total duration (minutes)	10.2±10.0	12.5±14.9	0.462
Mean age (days)	55.3±25.9	46.9±24.6	0.230
CSS	1.52±1.64	2.68±1.54	0.008
mDISCERN	1.16±1.17	3.50±1.22	<0.001
mJAMA	1.37±0.83	2.48±0.95	<0.001
Professional/government	0 (0)	7 (100)
Entertainment news	6 (60)	4 (40)	0.021*
Internet news	5 (63)	3 (60)	0.023*
Consumer	3 (33)	6 (67)	0.401*
Network news	5 (20)	25 (80)	0.837*
Newspaper	0 (0)	3 (100)	1.000*
Education	0 (0)	2 (100)	1.000*

Non-factual videos were found to have lower CSS (mean difference (MD): −1.15; 95% CI −2.04 to −0.26; p=0.008), mDISCERN (MD −2.34; 95% CI −2.99 to −1.69; p<0.001) and mJAMA (MD −1.11; 95% CI −1.61 to −0.62; p<0.001) scores compared with factual videos.

Bolded p values are significant (p<0.05).

Data are presented as (N (%)) or mean±SD.

*P values represent a comparison of the number of non-factual videos from each source to government and professional videos.

CSS, COVID-19 Specific Score; mDISCERN, modified DISCERN score; mJAMA, modified Journal of the American Medical Association.

Of the 19 non-factual videos, six were from entertainment news (32%), five were from network news (26%), five were from internet news (26%) and three were from consumer videos (13%). Within each video category, there was a greater proportion of non-factual videos in internet news (63%), entertainment news (60%), consumer videos (33%) and network news (20%), compared with 0% for professional and government videos. In addition, videos from internet news (MD 0.63; 95% CI 0.07 to 1.19; p=0.023) and entertainment news (MD 0.60; 95% CI 0.07 to 1.13; p=0.021) were significantly more likely to have non-factual information compared with professional and government videos.

Objective outcome measures of video content

As shown in table 2, professional and government videos demonstrated higher CSS than consumer videos (MD 2.21; 95% CI 0.10 to 4.32; p=0.037), mDISCERN scores than consumer videos (MD 2.46; 95% CI 0.50 to 4.42; p=0.008), internet news (MD 2.20; 95% CI 0.19 to 4.21; p=0.027) and entertainment news (MD 2.57; 95% CI 0.66 to 4.49; p=0.004), and mJAMA scores than entertainment news (MD 1.21; 95% CI 0.07 to 2.36; p=0.033) and consumer videos (MD 1.27; 95% CI 0.10 to 2.44; p=0.028).

Table 2

Demographic characteristics and comparison of the CSS, mDISCERN and mJAMA scores of each of the video sources to government and professional videos

Video source	Number	Views	CSS	mDISCERN	mJAMA
Government/professional	7 (10)	23 807 528 (9)	3.14±1.21	4.57±0.53	2.71±1.11
Entertainment news	10 (14)	54 244 095 (21)	2.50±2.27(p=0.694)	2.00±1.83(p=0.004)	1.50±0.85(p=0.033)
Internet news	8 (12)	30 269 699 (12)	2.00±1.77(p=0.327)	2.38±2.39(p=0.027)	1.63±1.41(p=0.089)
Network news	30 (43)	75 015 358 (29)	2.53±1.31(p=0.583)	3.13±1.19(p=0.104)	2.73±0.69(p=1.000)
Consumer	9 (13)	57 525 814 (22)	1.12±1.46(p=0.037)	2.12±1.25(p=0.008)	1.50±0.76(p=0.028)
Newspaper	3 (4)	13 034 084 (5)	1.33±2.30(p=0.241)	2.33±2.08(p=0.136)	2.00±1.00(p=0.700)
Education	2 (3)	3 907 568 (2)	3.50±2.12(p=1.000)	3.00±1.41(p=0.592)	1.00±0.00(p=0.081)

Bolded p values are significant (p<0.05).

Data are presented as (n (%)) or mean±SD.

CSS, COVID-19 Specific Score; MDISCERN, modified DISCERN score; mJAMA, modified Journal of the American Medical Association.

Descriptive analysis of non-factual statements

Examples of quotations of non-factual statements can be classified into four categories: statements related to the CSS criteria, recommendations for the general public, racist and discriminatory remarks and conspiracy theories, as shown in box 1.

Statements related to the CSS criteria:

‘Coronavirus only affects immunocompromised, cancer patients, and older people’.

‘The pharmaceutical companies have a cure but won't sell it so everyone is dying’.

‘A stronger strain of the virus is in Iran and Italy’.

Statements related to recommendations for the general public:

‘The only way to eat is to be violent’.

‘Some experts say to stock up with baby supplies and bottled water for months in case there is any disruption’.

Racist and discriminatory remarks

’Chinese virus’.

‘Chinese must understand it’s a sensitive time, do not misunderstand isolation as discrimination’.

Conspiracy theories

‘The world is controlled by a cult. This cult wants to control everyone. These people are the 1% and use an underground force to control people. This cult uses the mainstream media to tell premade versions of a story in order to incite fear and control in the public. Coronavirus is an example of one of these control tactics. This is done to control the economy to destroy small business’.

Discussion

This study evaluated the accuracy, usability and quality of the most widely viewed YouTube videos on COVID-19 in order to better understand the information viewed by the public on a dominant media platform. Our study highlights that over 25% of YouTube’s most viewed English videos contained non-factual or misleading information, reaching over 62 million views and nearly 25% of total viewership. There were high proportions of non-factual videos in the entertainment and internet news categories and moderate amounts in consumer and network news categories. In comparison with these sources, professional and government videos demonstrated higher accuracy, usability and quality across all measures but were largely under-represented in our sample. Our findings, which are consistent with those published in similar studies,8 10 11 suggest the lack of access of professional and government sources to individuals who use YouTube as a source of health information and a missed opportunity for dissemination of high-quality content.

In addition, descriptive analysis of non-factual videos included statements consisting of conspiracy theories, non-factual information, inappropriate recommendations inconsistent with current official government and health agency guidelines and discriminating statements. This is particularly alarming, when considering the immense viewership of these videos. Evidently, while the power of social media lies in the sheer volume and diversity of information being generated and spread, it has significant potential for harm. The proliferation and spread of misinformation can exacerbate racism and fear and result in unconstructive and dangerous behaviour,18 19 such as toilet paper hoarding and mask stealing behaviours seen so far in the COVID-19 pandemic. Consequently, this misinformation impedes the delivery of accurate pandemic-related information, thus hindering efforts by public health officials and healthcare professionals to fight the pandemic.18 19

The education and engagement of the public is paramount in the management of this pandemic by ensuring public understanding of and, therefore, adherence with public health measures. In current times, social media has emerged as a powerful tool in controlling the spread of information and in shaping public understanding and behaviour. YouTube is a powerful, untapped educational tool that should be better mobilised by health professionals to control information and influence public behaviour. Given that professionals and government organisations provide the highest quality content, these groups should find strategies to increase the viewership and the impact of their videos on YouTube. Many existing marketing strategies are static, in the form of published guidelines, statistical reports and infographics and may not be as appealing or accessible to the general public. Using multimedia and multiple social media platforms, including YouTube, will allow for greater impact. Furthermore, past research has demonstrated that using different emotional tones (eg, humour and anger) to deliver health information may increase public belief in such messages,20 which may consequently increase adherence. While members of governments and professionals already make appearances on network news programmes, partnerships between public health agencies and the creators of consumer and entertainment news videos may be an effective strategy to broaden their reach and impact. As an example, the most viewed video on YouTube at the time of this study, with over 20 million views, was produced by a popular YouTube and television celebrity.

There are several strengths of this study. We used a combination of objective validated tools as measures of reliability, specifically the mJAMA and mDISCERN criteria, in comparison with mainly descriptive studies published previously.8 11 Additionally, our sample included the top 69 widely viewed YouTube videos at the time of search, capturing over 250 million views and allowing for the analysis of video content accessed by many YouTube users. To our knowledge, this is the first study to report on the accuracy and quality of the most widely viewed YouTube videos and their publishing sources during the COVID-19 pandemic. As such, this study provides important insights into how professional organisations and health agencies can collaborate with content creators on YouTube to deliver high-quality information to a larger audience. Directing future public health efforts to mitigate the spread of misinformation may allow for increased public adherence with public health measures.

This study also has limitations to consider. First, this is a cross-sectional study at one time point. However, as YouTube is a dynamic platform, any search strategy would have this limitation. In addition, this limitation is unlikely to affect our conclusion given that they are supported by other studies at other time points.8 10 11 The search strategy, developed a priori, captured the top 75 videos for each of the two keywords; however, many of these videos were excluded per the reasons noted in figure 1, resulting in a final sample size of 69 videos. In future studies, the significant number of potential videos that may be excluded should be considered prior to data collection; however, this is difficult to predict ahead of time. Regardless, our sample size is larger than some other similarly designed studies on YouTube content published, such as Radonjic et al’s13 analysis of YouTube videos on abdominal aortic aneurysms (n=51 videos). Furthermore, the total viewership of our 69 included videos was 257 804 146 views and greatly surpasses the total viewership reported in other studies on diseases the WHO has declared as pandemics (H1N1: 5.1 million views for 142 videos8) or considered to be public health emergencies of international concern (Ebola in 2014: 9.1 million views for 118 videos9). Importantly, the goal of this present study was to assess the quality of the content that the most YouTube viewers are watching. We achieved this goal well with our sample size, methodology and total viewership of 257 804 146 views. The inclusion of only English videos presents a language bias and may limit the generalisability of our results to different languages and countries. As the literature lacks a validated tool to evaluate COVID-19 related content, we developed an unvalidated but objective COVID-19 specific scoring system to assess the videos included in our study. In addition, our search was limited to the YouTube platform and, consequently, YouTube videos present on other internet sites were not captured.

Future studies should examine YouTube content on COVID-19 as the pandemic progresses and YouTube content in other languages than English. Furthermore, studies to determine the propagation of YouTube videos onto other social media platforms such as Instagram, Twitter, Facebook and so on will lend additional insights into their impact. This information will provide public health officials with a better understanding of the sources of information the public is using to learn about the current COVID-19 pandemic and direct efforts to educate the public in future public health emergencies.

Conclusion

Among the most viewed English videos regarding COVID-19 on YouTube, 27.5% contained non-factual information originating from entertainment news, internet news and consumer sources, reaching 62 million views worldwide. Videos from professional and government organisations were the most informative and had the highest quality content but were greatly under-represented in terms of viewership. As YouTube is already a dominant media platform, we recommend that public health agencies collaborate with a wider range of YouTube producers (eg, entertainment news, internet news and influential consumers) to disseminate high-quality video content. This will be an effective and immediately implementable public health strategy to effectively capture a wider audience from all demographic backgrounds, thus educating the public and minimising the spread of misinformation. Especially considering the worsening progression of the current COVID-19 pandemic, this will likely play a significant role in increasing public adherence with public health measures and in the collective fight against the COVID-19 pandemic.