YouTube as a source of misinformation on COVID-19 vaccination: a systematic analysis.

INTRODUCTION: Vaccines for SARS-CoV-2 have been accessible to the public since December 2020. However, only 58.3% of Americans are fully vaccinated as of 5 November 2021. Numerous studies have supported YouTube as a source of both reliable and misleading information during the COVID-19 pandemic. Misinformation regarding the safety and efficacy of COVID-19 vaccines has negatively impacted vaccination intent. To date, the literature lacks a systematic evaluation of YouTube's content on COVID-19 vaccination using validated scoring tools. The objective of this study was to evaluate the accuracy, usability and quality of the most widely viewed YouTube videos on COVID-19 vaccination.
METHODS: A search on YouTube was performed on 21 July 2021, using keywords 'COVID-19 vaccine' on a cleared-cache web browser. Search results were sorted by 'views', and the top 150 most-viewed videos were collected and analysed. Duplicate, non-English, non-audiovisual, exceeding 1-hour duration, or videos unrelated to COVID-19 vaccine were excluded. The primary outcome was usability and reliability of videos, analysed using the modified DISCERN (mDISCERN) score, the modified Journal of the American Medical Association (mJAMA) score and the COVID-19 Vaccine Score (CVS).
RESULTS:
CONCLUSION: As the COVID-19 pandemic evolves, widespread adoption of vaccination is essential in reducing morbidity, mortality, and returning to some semblance of normalcy. Providing high-quality and engaging health information from reputable sources is essential in addressing vaccine hesitancy. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Throughout the COVID-19 pandemic, videos on social media questioning the efficacy and safety of COVID-19 vaccines have become more numerous and widespread. It has previously been reported that 27.5% of the most viewed YouTube videos on COVID-19 contained misinformation.

Information from reputable sources, such as the WHO or the Centers for Disease Control and Prevention, is under-represented on social media.

Approximately 11% of YouTube’s most viewed videos on COVID-19 vaccines, accounting for over 18 million views, contradicted the reference standard from the WHO, or the Centers for Disease Control and Prevention.

Information from reputable sources may be perceived as less favourable by the public, when proxied by the ratio of likes to dislikes. On average, YouTube videos from government sources had triple the ratio of dislikes to likes, while entertainment videos had 14 times the ratio of likes to dislikes.

Despite efforts by YouTube in combatting health misinformation, a substantial portion of highly viewed content contains misinformation regarding COVID-19 vaccines continues to be hosted by the second-largest social media platform.

Public health officials and clinicians should recognise the impact of YouTube and other social media platforms as sources of health information and misinformation for patients and leverage these platforms to disseminate high-quality content to combat misinformation given poor vaccination uptake.

Introduction

COVID-19, caused by the novel SARS-CoV-2, was first noted in December 2019 and has evolved into a global pandemic with over 246 million cases worldwide and nearly 5 million deaths to date.1 Vaccines against SARS-CoV-2 became accessible to the public since December 2020, after being granted emergency use authorisation status by the Federal Drug Administration.2 Despite this, as of 5 November 2021, only 58.3% of Americans are fully vaccinated.3 This is concerning as public trust in vaccination programmes is paramount in reducing morbidity and mortality, limiting the spread of highly transmissible SARS-CoV-2 variants, and putting an end to this pandemic.4

In the digital age, more individuals are relying on the internet and social media as sources of healthcare-related information.5–8 In particular, YouTube is one of the most predominant social media platforms with billions of views daily and is the second most used search engine after Google.9 This readily accessible platform offers multiple genres of content, including entertainment, news, consumers, and education, and allows its users to interact with each other in real time.

Numerous studies have supported YouTube as a source of both reliable and misleading information during the COVID-19 pandemic.10–12 A previous study demonstrated that 27.5% of the most viewed YouTube videos on COVID-19 contained misinformation.11 As vaccine misinformation has been shown to negatively impact vaccination intent,12 propagation of misinformation regarding the safety and efficacy of COVID-19 vaccines on large platforms, such as YouTube, has potential devastating consequences on public health.

To date, the literature lacks a systematic evaluation of YouTube as a source of information on COVID-19 vaccination using validated scoring tools. The objective of this study is to evaluate the accuracy, usability and quality of the most widely viewed YouTube videos on COVID-19 vaccination.

Methods

Search protocol

A YouTube search was performed on 21 July 2021, using keywords ‘COVID-19 vaccine’ on a cleared-cache web browser. Search results were sorted by ‘views’, and 150 videos with the greatest number of views were collected and analysed.

Study selection and data extraction

Three reviewers (HO-YL, EP and OB-L) reviewed and analysed the 150 videos. Videos that were duplicate, non-English, non-audio, non-visual or unrelated to COVID-19 vaccine were excluded. Discrepancies were resolved by discussion for consensus.13 Video descriptive characteristics including title, hyperlink, number of views, number of likes and dislikes, upload date, duration and publishing category, were collected.

The reliability and quality of video content were assessed using the modified DISCERN (mDISCERN) score (online supplemental table S1) and the modified Journal of the American Medical Association (mJAMA) criteria (online supplemental table S2).14 15 A novel five-point COVID-19 Vaccine Score (CVS), developed based on previously published work, was used to evaluate the usefulness of video content for the average viewer (online supplemental table S3).16

Patient and public involvement

No patients or public were involved in any way in this study.

Data analysis

Following data extraction, two independent reviewers (EP and OB-L) categorised all videos into videos containing factual or non-factual information, and into their video publishing category, in duplicate, with a third reviewer (HO-YL) consultation for consensus. Video sources were categorised into healthcare professionals, network news, education, internet news, consumers, entertainment, government sources and newspapers according to previously published work (online supplemental table S4).17 Videos were considered non-factual if they contained one or more non-factual statements, defined as contradictory to guidelines published by public health agencies (ie, WHO or Centers for Disease Control and Prevention) at the time of video publication.18 19 The CVS score was calculated as follows: if a CVS criterion was discussed, it was given a score of 1 if factually correct, a score of 0.5 if ambiguous and 0 if non-factual (online supplemental table S3). The total score was then divided by the number of items discussed (up to a maximum of 5) and expressed as a proportion. mDISCERN and mJAMA scores were calculated with one point given for each criterion correctly addressed (online supplemental tables S1 and S2).

Descriptive statistics including two tailed t-tests and Mann-Whitney U tests compared factual videos to non-factual videos, with a significance level of p<0.01 to offset multiple hypothesis testing. Percentages, means and ratios compared videos from different sources. All analyses were performed using Microsoft Excel and Mathworks Matlab software (V.R2019b).

Results

Video characteristics

Of 150 videos screened, 122 videos (81%) were included. A total of 28 (19%) videos were excluded: 14 for not discussing any CVS criteria, 13 for non-English and one for non-audio (online supplemental figure S1).

Source of videos

Videos were classified into eight categories (figure 1). The total viewership at time of data collection was 169 446 382 views. 24.1% of views were from videos produced by healthcare professionals, 20.9% from network news, 16.9% from education, 10.2% from internet news, 8.4% from consumers, 8.1% from entertainment, 7.3% from government sources and 4.2% from newspapers.

Figure 1

Number of included videos by publishing source.

Government videos had the highest average CVS score (0.98) and number of CVS items discussed per minute of video (1.89 items/min). Internet news videos had the lowest CVS score (0.83), and consumer videos had the lowest CVS items discussed per minute (0.20 items/min). Internet news also had the lowest mDISCERN and mJAMA scores, whereas government videos had the highest mJAMA scores and educational videos had the highest mDISCERN scores (table 1).

Table 1

Comparison of videos by source category

Channel type	% of total views	CVS	mDISCERN	mJAMA	Age (days)	Median likes/dislikes (range)	Mean subscribers (M)	Mean views (M)
Health professional n=25 (20.5%)	24.11	0.93 (0.15)	3.7 (0.95)	1.2 (0.66)	116 (75)	2.8 (0.16–22.2)	0.62	1.63
Entertainment n=7 (5.7%)	8.10	0.95 (0.96)	3.6 (1.40)	1.3 (0.95)	127 (34)	14.4 (0.91–52.2)	6.58	1.96
Network news n=42 (34.4%)	20.92	0.90 (0.18)	3.0 (1.23)	1.0 (0.68)	99 (85)	0.72 (0.15–20.2)	4.26	0.83
Internet news n=6 (4.9%)	10.16	0.83 (0.20)	2.0 (2.0)	0.5 (0.84)	103 (64)	15.4 (0.30–23.2)	3.56	2.87
Government n=9 (7.4%)	7.26	0.98 (0.07)	3.8 (0.97)	1.3 (0.87)	65 (57)	0.32 (0.10–2.96)	0.57	1.37
Newspaper n=4 (3.3%)	4.22	0.84 (0.20)	3.8 (0.96)	1.3 (0.96)	310 (180)	1.58 (0.71–13.9)	1.71	1.79
Consumer n=10 (8.2%)	8.38	0.88 (0.20)	3.5 (1.65)	1.1 (0.74)	223 (157)	3.0 (1.07–22.8)	0.91	1.42
Education n=19 (15.6%)	16.85	0.98 (0.05)	4.3 (1.11)	1.2 (1.17)	167 (78)	9.7 (1.23–35.8)	3.11	1.50

Mean (SD), unless otherwise indicated. M represents millions.

CVS, COVID-19 Vaccine Score; mDISCERN, modified DISCERN; mJAMA, modified Journal of the American Medical Association.

Government videos had the shortest average time since publication of 65 days, compared with 310 days for newspaper videos. Entertainment videos on average had 14.4 times more likes than dislikes, whereas government videos averaged more dislikes than likes, with 3.1 dislikes to every like.

Videos with factual versus non-factual information

Thirteen videos (10.7%) contained non-factual information and accounted for 11.0% of total viewership, while 109 videos (89.3%) contained only factual information. Of the 13 non-factual videos, 4 were from network news, three from healthcare professionals, two each from education and internet news, and one each from newspaper and consumer sources. Two of the 13 videos (15.4%) containing non-factual information did not contain a YouTube information icon prompting the viewer to seek additional information on COVID-19 vaccination.

When comparing the two groups, there were no significant differences in views per video (p=0.94), likes per video (p=0.40), dislikes per video (p=0.36), duration (p=0.98) and mean days since publication (p=0.56) (table 2). Videos with non-factual information on average had a greater median ratio of likes to dislikes, compared with videos with factual information (13.89 vs 2.33, p<0.01).

Table 2

Comparison of factual and non-factual videos

	Non-factual	Factual	P value
No of videos	13 (10.66%)	109 (89.34%)
No of views	18 482 373 (11%)	150 964 009 (89%)
Views per video (in millions)	1.421 (1. 120)	1.385 (1.677)	0.939
CVS	0.7 (0.202)	0.948 (0.122)	<0.00001
mDISCERN	1.769 (1.235)	3.651 (1.181)	<0.00001
mJAMA	0.538 (0.66)	1.183 (0.807)	0.00649
Median likes/dislikes (range)	13.89 (0.49–22.8)	2.33 (0.10–52.2)*	0.00078†
No of comments	9214 (9693)	6308 (9920)	0.319
No of days since publication	144.9 (114.1)	127.6 (100.3)	0.564
No of likes	25 539 (19945)	18 815 (33716)	0.398
No of dislikes	3400.7 (5402.1)	5668.2 (8596.5)	0.365
Average duration (min)	8.38 (6.10)	8.45 (10.59)	0.981
Category
Health professional	3 (23.1%)	22 (20.2%)
Entertainment	0 (0%)	7 (6.4%)
Network news	4 (30.7%)	38 (34.9%)
Internet news	2 (15.4%)	4 (3.7%)
Government	0 (0%)	9 (8.3%)
Newspaper	1 (7.7%)	3 (2.8%)
Consumer	1 (7.7%)	9 (8.3%)
Education	2 (15.4%)	17 (15.6%)

Data are represented as n (%) or mean (SD). P value for two-tailed t-test is shown where applicable. A significance threshold of p<0.01 was used.

Significant p-values highlighted in bold.

*n=101 as 8 videos hid their likes and dislikes count.

†Mann-Whitney U test used in lieu of t-test.

CVS, COVID-19 Vaccine Score; mDISCERN, modified DISCERN; mJAMA, modified Journal of the American Medical Association.

Videos with non-factual information were found to have lower CVS (p<0.001), mDISCERN (p<0.001) and mJAMA (p<0.01) scores, compared with videos with factual information. Classification and examples of non-factual statements are shown in table 3.

Table 3

Non-factual claims made in YouTube videos about COVID-19 vaccination*

Reference standard	No of videos containing non-factual statements that contradict reference standard	Examples of non-factual statements
COVID-19 is a serious disease	1	‘Immunity (natural and through healthy habits) and vitamin D are enough to survive COVID-19’‘This virus is like pollen in the air…God is the real healer.’
COVID-19 vaccine is effective	3	‘Right now we are in the middle of a storm, and a vaccine is not going to help.’‘I had COVID-19 so I don't need the vaccine’
COVID-19 vaccine is generally safe	6	‘I have autoimmune disease so I can’t have it’‘Vaccine affects fertility’
There are very few contraindications to the COVID-19 vaccine	3	‘Those who have any history of …. fever (may be due to any other cause), bleeding disorder, on blood thinners or immunocompromised…. or anyone suffering from serious health related issues…must not take vaccines’‘Old people should avoid getting vaccinated’

*https://www.who.int/emergencies/diseases/novel-coronavirus-2019/covid-19-vaccines.

Discussion

This study highlights that approximately 11% of YouTube’s most viewed videos on COVID-19 vaccines contradicted the reference standard,18 19 reaching over 18 million viewers. Our findings were significantly lower than other studies reporting 27.5% and 45% of videos containing non-factual information.11 16 This discrepancy is likely due to increased awareness of the impact of social media on disseminating misinformation that has resulted in YouTube’s implementation of a COVID-19 medical misinformation policy in May 2020.20 Furthermore, government sources are increasingly leveraging YouTube as a valuable opportunity to disseminate vaccine information, with an over twofold increase in the percentage of government videos in this study compared with a study published earlier in the pandemic on COVID-19 misinformation.11 Despite producing the most reliable content, videos from government sources had over three times the ratio of dislikes to likes. In comparison, entertainment videos had a ratio of over 14 times the number of likes to dislikes. Similarly, the median ratio of likes to dislikes for videos containing non-factual information was almost six times higher than that of videos containing factual information. This suggests the presence of an active community of users who are sceptical of the COVID-19 vaccine and government sources. It is important to note that this may also represent a selection bias, as a previous study demonstrated that participants who relied on private sources for COVID-19 information, such as network news and social media, were less knowledgeable about COVID-19 and less likely to adhere to social distancing guidelines.21 Additionally, another study demonstrated that YouTube users were less likely to receive the COVID-19 vaccine compared with non-users.22

Developing timely strategies to monitor and optimise the quality of information on YouTube during the COVID-19 pandemic is crucial for two reasons. First, it may be difficult for the YouTube algorithm to discern the accuracy and reliability of information conveyed. For example, the infamous COVID-19 film, Plandemic, which contained significant misinformation and conspiracy theories, was only removed from YouTube after public outcry, but not before it had already garnered millions of views at the time of removal, with the potential for serious consequences.23 Second, the YouTube algorithm allegedly makes recommendations for videos with a higher number of views and a greater proportion of likes.24 25 Moreover, the algorithm also allegedly adjusts video recommendations based on the user’s history of videos watched.25 Thus, it is plausible that individuals who encounter an initial video containing misinformation, may continue to receive recommendations for other videos containing similar misinformation. To address these concerns, in 2020, YouTube implemented an automated flagging system to demonetise and remove videos containing COVID-19 misinformation.26 However, this system’s reliance on automation has resulted in enforcement errors, such as mistakenly removing videos which sought to refute COVID-19 misinformation.27 Potential improvements could include increasing viewership of reputable government sources, as they have been shown to be the most reliable, by suggesting these videos to users. Furthermore, given that over 85% of those aged 13–17 years old use YouTube,28 algorithms suggesting youth-friendly government videos on COVID-19 vaccination to this audience may help increase vaccine compliance in the younger population, which has been reported to be much lower than among adults.29

Conclusion

As the COVID-19 pandemic evolves, widespread vaccination is essential in reducing morbidity, mortality and returning to some semblance of normalcy. Providing high-quality health information from reputable sources that is engaging to a variety of viewers will be essential in addressing vaccine hesitancy and improving the public’s understanding and adherence to public health measures. Policy-makers should consider implementing mandates surrounding medical misinformation on social media platforms. Public health officials and clinicians should recognise the impact of YouTube and other similar social media sites as sources of health information for patients from all demographic backgrounds and leverage these platforms to disseminate high-quality content to combat misinformation.