Assessing the quality of COVID-19 vaccine videos on video-sharing platforms.

BACKGROUND: Video-sharing platforms are a common source for health information such as Coronavirus Disease 2019 (COVID-19) vaccines. It is important that they provide good quality, evidence-based information. However, to date, the quality of information surrounding COVID-19 vaccines on video-sharing platforms has not been established.
OBJECTIVE: This study developed an assessment tool to evaluate the quality of Coronavirus Disease 2019 (COVID-19) vaccine videos on YouTube, Facebook Watch and TikTok.
METHODS: Assessment of quality was based on understandability, actionability, accuracy, comprehensiveness and reliability. Videos were searched using the keywords "COVID-19 vaccine", "Coronavirus vaccine" and "SARS-CoV-2 vaccine". Seventy-two videos were evaluated. Descriptive statistics, Kruskal-Wallis and Wilcoxon-rank sum tests were used for analysis.
RESULTS: YouTube had the highest median composite score compared to TikTok (36.8% versus 27.5%, p = 0.001). YouTube also had the highest median reliability score (37.5%), but those of Facebook Watch (35.0%) and TikTok (35.0%) were only marginally lower. Median accuracy scores of all platforms were 100%, but their median comprehensiveness scores were low (YouTube 12.5%; Facebook Watch 6.3%; TikTok 6.3%, p = 0.004). Median actionability scores (0%) were the lowest for all platforms. TikTok had the highest median understandability score compared to YouTube and Facebook Watch (96.9% versus 80.0 each, p < 0.001).
CONCLUSION: The overall quality of videos on all video-sharing platforms were low. All platforms provided accurate COVID-19 vaccine information, but TikTok videos were the most understandable. Most videos did not provide full details about COVID-19 vaccines, thus viewers would need to watch several videos before making a better-informed decision.

Introduction

The Coronavirus Disease 2019 (COVID-19) pandemic has led to more than 100 million cases worldwide and over 2 million deaths. In order to slow down the spread of the disease, manufacturers have been striving to produce a vaccine since January 2020. As of June 2021, there were 16 vaccines approved for use (Appendix 1).

Lockdown was implemented in many countries in the attempt to stop the spread of the coronavirus. As people were restricted to their homes for extended periods, the need to keep updated with the news on the pandemic led to an increase in social media usage. Video-sharing platforms like YouTube, Facebook Watch (Facebook's video platform) and TikTok consistently ranked highly as information sources for individuals to obtain news on COVID-19, including vaccine-related information. In October 2020, YouTube, Facebook Watch and TikTok attracted 2 billion, 1.25 billion, and 690 million monthly active users respectively. Unfortunately, the infodemic led to misinformation being spread on these video-sharing platforms.

While previous studies have assessed the quality of information regarding the COVID-19 pandemic on video-sharing platforms and on the internet, from our knowledge, there are currently no studies which have investigated the quality of information surrounding COVID-19 vaccines on video-sharing platforms. There is a need to ascertain the quality of information on these platforms, as misinformation is associated with vaccine hesitancy, which can impact one's health, and more broadly, the ability to achieve herd immunity.

Multiple quality assessment tools have been developed to evaluate the quality of health information on the internet and on videos (Table 1). While these tools are useful for evaluating health websites, they differ slightly in their evaluation criteria. At present, tools used for evaluating videos are limited in scope as they either concentrate on evaluating information about specific health conditions,, or the characteristics and understandability of a video., Among the current evaluation tools, only the COVID-19 Specific Score (CSS), Medical Information and Content Index (MICI) and Video Information and Quality Index (VIQI) contain criteria to evaluate the accuracy of information provided in the videos. However, these evaluation criteria are simplistic and do not provide clear guidance as to what would be considered accurate information., Furthermore, while the CSS evaluates COVID-19 information, it addresses topics regarding the pandemic itself such as the coronavirus' epidemiology and transmission, thus it cannot be applied to information on COVID-19 vaccines. A common limitation across the tools is that none can assess the comprehensiveness of information. To our knowledge, there is currently no quality assessment tool which evaluates the quality of videos on COVID-19 vaccines. Hence, the objective of this study was to develop a quality assessment tool to evaluate the accuracy, comprehensiveness, reliability, understandability and actionability of COVID-19 vaccine videos on three common video-sharing platforms (YouTube, Facebook Watch and TikTok).

Table 1

Summary of quality evaluation tools for assessing online health information and videos.

	HONcode	JAMA	DISCERN instrument	LIDA	QUEST	QCSS	MICI	CSS	Usefulness Score	Customised usefulness score	VIQI	PEMAT A/V
	https://www.hon.ch/cgi-bin/HONcode/principles.pl?English	https://pubmed.ncbi.nlm.nih.gov/9103351/	http://www.discern.org.uk/discern_instrument.php	http://www.minervation.com/wp-content/uploads/2011/04/Minervation-LIDA-instrument-v1-2.pdf	https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6194721/	https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1772623/	https://www.ncbi. nlm.nih.gov/pmc/articles/PMC4550615/	https://gh.bmj.com/content/5/5/e002604	https://www.wjgnet.com/1007-9327/full/v20/i14/4066.htm#T1%20	https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6636170/	https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4550615/	https://www.ahrq.gov/health-literacy/patient-education/pemat-av.html
	Evaluation tools for online health information (internet)						Evaluation tools for videos
							Disease-specific				Non-disease specific
Reliability
Authorship (Provides author name and qualification)	✓	✓	✗	✗	✓	✓	✗	✗	✗	✗	✗	✗
Complementarity (Supports, not replace the role of a physician)	✓	✗	✓	✗	✓	✗	✗	✗	✗	✗	✗	✗
Confidentiality (Respects user privacy)	✓	✗	✗	✗	✗	✗	✗	✗	✗	✗	✗	✗
Attribution of sources (Sources are cited)	✓	✓	✓	✓	✓	✓	✗	✗	✗	✗	✗	✗
Justifiability (Balanced and objective claims)	✓	✗	✓	✗	✓	✗	✗	✗	✗	✗	✗	✗
Transparency (Provide contact details)	✓	✗	✗	✗	✗	✓	✗	✗	✗	✗	✗	✗
Financial disclosure (Funding details are provided)	✓	✓	✗	✓	✗	✓	✗	✗	✗	✗	✗	✗
Advertising (Distinguish advertising and editorial content)	✓	✗	✗	✗	✓	✓	✗	✗	✗	✗	✗	✗
Currency of content (Dates of the information cited are provided)	✗	✓	✓	✓	✓	✓	✗	✗	✗	✗	✗	✗
Purpose of the site (What the site is about and what it is meant to cover)	✗	✗	✓	✓	✗	✓	✗	✗	✗	✗	✗	✗
Links to other resources (Provide additional sources of information)	✗	✗	✓	✗	✗	✗	✗	✗	✗	✗	✗	✗
Relevance (Is the information relevant to the user)	✗	✗	✓	✗	✗	✗	✗	✗	✗	✗	✗	✗
User's opinion of the overall quality of the publication	✗	✗	✓	✗	✗	✗	✗	✗	✗	✗	✗	✗
Address areas of uncertainty	✗	✗	✓	✗	✗	✗	✗	✗	✗	✗	✗	✗
Accessibility of site (Is the information accessible to those who need it)	✗	✗	✗	✓	✗	✗	✗	✗	✗	✗	✗	✗
Usability (Can users make sense of the site)	✗	✗	✗	✓	✗	✗	✗	✗	✗	✗	✗	✗
Accuracy
Prevalence of disease	✗	✗	✗	✗	✗	✗	✓	✓	✗	✓	✗	✗
Transmission of disease	✗	✗	✗	✗	✗	✗	✓	✓	✗	✗	✗	✗
Signs & Symptoms of disease	✗	✗	✗	✗	✗	✗	✓	✓	✓	✓	✗	✗
Screening Testing	✗	✗	✗	✗	✗	✗	✓	✗	✗	✗	✗	✗
Treatment/Outcome	✗	✗	✗	✗	✗	✗	✓	✓	✓	✓	✗	✗
Cause of condition	✗	✗	✗	✗	✗	✗	✗	✗	✓	✓	✗	✗
Diagnosis of condition	✗	✗	✗	✗	✗	✗	✗	✗	✓	✓	✗	✗
Recovery from condition	✗	✗	✗	✗	✗	✗	✗	✗	✓	✗	✗	✗
Risk Factors	✗	✗	✗	✗	✗	✗	✗	✗	✗	✓	✗	✗
Prevention	✗	✗	✗	✗	✗	✗	✗	✗	✗	✓	✗	✗
Information accuracy of video	✗	✗	✗	✗	✗	✗	✗	✓	✗	✗	✓	✗
Understandability and Actionability
Understandability (Degree that users can explain the content of the video)	✗	✗	✗	✗	✗	✗	✗	✗	✗	✗	✓	✓
Actionability (Whether users can identify actions to take after watching the video)	✗	✗	✗	✗	✗	✗	✗	✗	✗	✗	✗	✓
Precision (level of coherence between video title and content)	✗	✗	✗	✗	✗	✗	✗	✗	✗	✗	✓	✗

✓ Criterion is evaluated by the tool, ✗ Criterion is not evaluated by the tool.

HONcode: Health On the Net code; JAMA: Journal of American Medical Association benchmark; LIDA: Minervation Validation Instrument; QUEST: Quality Evaluation Scoring Tool; QCSS: Quality Component Scoring System; MICI: Medical Information Content Index; CSS: COVID-19 Specific Score; VIQI: Video Information and Quality Index; PEMAT A/V: Patient Education Materials Assessment Tool for Audio-Visual Materials

Methods

Video selection

Videos were searched on YouTube, Facebook Watch and TikTok on 2nd March 2021 using the keywords “COVID-19 vaccine”, “Coronavirus vaccine”, and “SARS-CoV-2 vaccine”. YouTube and Facebook Watch were accessed using an incognito Google Chrome browser with a cleared cache and without any login. Likewise, TikTok searches were conducted on a newly installed mobile app on an iPhone 12 (v.14.4) with a cleared cache and without any login. The same process was applied to an Android phone (Samsung Note 20 Ultra 5G, Android 11), which yielded the same search results. Therefore, only the iPhone video results were used for evaluation. These measures were applied to ensure that the videos were not recommended based on the reviewers' previous searches. Searches were conducted using the default “by relevance” sorting system on all platforms and without any filters, in order to closely reflect the way users typically search for information. The first 20 results of each keyword search were selected from each of the three platforms, based on previous statistics which indicate that most users do not go past the first three pages of results on a search engine, and analysis of click-through rates (ratio of users who click on a link compared to the total number of users who view the page). The Uniform Resource Locators (URLs) for the videos were saved as backup.

Videos in English and on currently approved COVID-19 vaccines were included (Fig. 1). Non-English videos, those unrelated to COVID-19 vaccines and duplicate videos were excluded. Additionally, videos longer than 12 mins were excluded due to a drop in engagement in longer videos. Video parameters extracted were the total number of views, video length, number of likes, type of institution/individual who uploaded the video, and upload date. Videos were categorised into three tiers of author qualification grading – Tier One videos were by authoritative sources with recognised credentials whose advice could be used to guide healthcare decisions; Tier Two included videos in which the author had some level of healthcare/public health expertise; and Tier Three were videos from independent channels or news agencies whose credentials would not be recognised in the healthcare/public health sphere, or if no information was provided about the video's authorship. All videos were categorised as “general” (addressing COVID-19 vaccines as a whole) and “specific” (focusing on particular vaccine brands or types).

Fig. 1

Search methodology for YouTube, Facebook Watch and TikTok videos.

Quality assessment

The quality parameters used for evaluation of the videos were understandability, actionability, accuracy, comprehensiveness and reliability (Table 2). The Patient Education Materials Assessment Tool for Audio-Visual Materials (PEMAT A/V) tool was adopted to evaluate understandability and actionability. Each criterion was scored on a 2-point agreement scale (0 = Disagree, 1 = Agree). The accuracy section evaluated the following aspects – preventive measures pre- and post-vaccination, population groups who should be vaccinated, and COVID-19 vaccine information (expedited approval of vaccines, vaccine types, administration, side effects and benefits). Each criterion was scored on a 3-point accuracy scale (0 = all points inaccurate, 1 = points partially accurate, 2 = all points accurate). Comprehensiveness was evaluated using a 3-point comprehensiveness scale (0 = lesser than or equal to 33% of points mentioned, 1 = 34–67% of points mentioned, 2 = 68% or more of points mentioned). The COVID-19 vaccine information used for evaluation were obtained and collated from the COVID-19 frequently asked questions (FAQ) sections of the websites from the World Health Organization (WHO), US Centers for Disease Control and Prevention (CDC), Singapore Ministry of Health (MOH), UK National Health Service (NHS), and the Australian Government Department of Health. Criteria that did not apply to the videos were classified as “Not Applicable” (NA).

Table 2

Definition of the quality parameters used in this study.

Quality Parameter	Definition	Measurement Component/Score	Adapted from
Understandability	Consumers of diverse backgrounds and varying levels of health literacy can process and explain the key messages of the videos	0 = Disagree, 1 = Agree	Patient Education Materials Assessment Tool for Audio-visual Materials (PEMAT A/V)
Actionability	Consumers of diverse backgrounds and varying levels of health literacy can identify what they can do based on the information presented in the videos
Accuracy	Information in the videos is “scientifically correct”	0 = all points inaccurate,1 = points partially accurate,2 = all points accurate	Criteria based on FAQs from the World Health Organization (WHO), US Centers for Disease Control and Prevention (CDC), Singapore Ministry of Health (MOH), UK National Health Service (NHS), and the Australian Government Department of Health
Comprehensiveness	The extensiveness in which the criteria obtained from the Frequently Asked Questions (FAQs) are described in the videos	0 = lesser than or equal to 33% of points mentioned,1 = 34–67% of points mentioned,2 = 68% or more of points mentioned
Reliability	The extent of trustworthiness of the videos as a source of information on COVID-19 vaccines	2 or 3-point Likert scale, with a higher score translating to a better reliability	DISCERN Instrument and Quality Component Scoring System

The DISCERN and Quality Component Scoring System (QCSS) tools were adapted to evaluate reliability. Criteria included the attribution of information sources, biasness of the videos, currency of information sources, external sources of support and information, and references to areas of uncertainty about treatment. Additional criteria from QCSS evaluated authorship, author qualification grading, sponsorship disclosure, transparency, statement of purpose and commercial agenda of the videos. Statement of purpose and commercial agenda were scored on a 2-point Likert scale. The other criteria were scored on 3-point Likert scales customised for each criterion (Appendix 2).

The initial version of the quality assessment tool was piloted by evaluating five English COVID-19 vaccine videos on WeChat – which is China's most popular multi-purpose messaging platform. A separate platform was used for this pilot to avoid influencing the videos that would be evaluated on the three platforms in this study. Feedback from the reviewers was consolidated to develop the final version of the tool, so as to ensure consistency in ratings in this study.

Two independent reviewers (RT and AP) evaluated the videos. A third reviewer (KY or LW) helped to resolve any discrepancies between the two reviewers. The scores for each domain of understandability, actionability, accuracy, comprehensiveness and reliability were reported as percentages and calculated by taking the sum of points that each video scored (sum of total points), divided by the sum of points that each video could have potentially scored if it achieved a full score (total possible points). Similarly, the composite score for each video was also calculated by taking the sum of points scored across all quality domains, divided by the sum of points the video could have potentially scored from all quality domains. Final scores used for analysis were calculated by taking the average of the two reviewers' scores. For understandability and actionability, a score of 70% or below indicated that the video was poorly understandable or actionable.

Statistical analysis

Data analysis was carried out using the Statistical Package for Social Sciences (SPSS, version 26.0). Medians and interquartile ranges (IQRs) were used to describe the video parameters (Fig. 1), quality domain scores and the composite scores of the videos. Shapiro-Wilk test was used to assess the normality of data. Kruskal-Wallis test was conducted to compare the scores of the quality domains and composite scores of the three platforms, and to compare the scores based on author qualification gradings. Wilcoxon rank sum test was used to compare the scores of the “general” and “specific” videos, and as post-hoc analyses of the quality domain and composite scores. Kendall's coefficient of concordance was used to evaluate inter-rater reliability for each domain and the composite scores. A two-tailed p-value of <0.05 was considered to be statistically significant.

Results

Seventy-two videos were uploaded between March 2020 and March 2021. Half (n = 35, 48.6%) were from YouTube, one-third (n = 23, 31.9%) from Facebook Watch and one-fifth (n = 14, 19.4%) from TikTok. Two videos (2.8%) were uploaded before the approval of the first vaccine (11th August 2020). Facebook Watch had the highest number of views (median = 428,151, IQR = 23,857–1,948,694) and likes (median = 2,400, IQR = 201–103,500), while YouTube had the lowest viewership (median = 93,137, IQR = 7,786–358,379) and likes (median = 737, IQR = 51–5,500) (Table 3). In contrast, YouTube had the longest video durations (median = 4.2, IQR = 2.7–6.8 mins), compared to Facebook Watch (median = 1.7, IQR = 1.3–2.7 mins, p < 0.001) and TikTok (median = 0.9, IQR = 0.6–1.0 mins, p < 0.001).

Table 3

Video parameters of the videos on YouTube, Facebook Watch and TikTok.

	Median number of views (IQR)	Median number of Likes (IQR)	Median Duration (mins) (IQR)	Date of Upload range
YouTube (n = 35)	93,137 (7,786–358,379)	737 (51–5,500)	4.2 (2.7–6.8)	15 Mar 2020–10 Feb 2021
Facebook Watch (n = 23)	428,151 (23,857–1,948,694)	2,400 (201–103,500)	1.7 (1.3–2.7)	03 Dec 2020–28 Feb 2021
TikTok (n = 14)	96,000 (4,610–682,225)	1,848 (438–43,113)	0.9 (0.6–1.0)	12 Nov 2020–02 Mar 2021
Total (N = 72)	139,603 (7,786–789,228)	1,264 (94–11,300)	2.3 (1.2–5.1)	15 Mar 2020–02 Mar 2021
p-value	0.29	0.18	<0.001*, a	–

p < 0.05 based on Kruskal-Wallis test.

Median durations between YouTube and Facebook Watch (p < 0.001), YouTube and TikTok (p < 0.001) and Facebook Watch and TikTok (p < 0.001) were statistically significant based on Wilcoxon Rank Sum test with Bonferroni adjustment.

Among the video-sharing platforms, the composite quality scores of YouTube (median = 36.8%, IQR = 30.7–43.0%) were significantly higher than TikTok (median = 27.5%, IQR = 24.7–31.5%, p = 0.001) (Table 4). In relation to the quality domains, accuracy scores were the highest (median = 100% for all platforms), while actionability scores were the lowest (median = 0% for all platforms). The second highest scoring quality domain was understandability. Median understandability scores of TikTok (96.9%) were significantly higher than YouTube and Facebook Watch (80.0% each, p < 0.001). On the other hand, median reliability (35.0% each) and comprehensiveness scores (6.3% each) were the same for Facebook Watch and TikTok. YouTube was the highest scoring platform in these two domains (median = 12.5% for comprehensiveness, p = 0.004; median = 37.5% for reliability, p = 0.078). Inter-rater agreement for all quality domains were high. The lowest scoring inter-rater domain was accuracy (W = 0.880).

Table 4

Median percentage scores for each quality domain and composite scores among video-sharing platforms, videos with different author qualification grading, and “general” and “specific” videos.

	Median Accuracy Score % (IQR)	Median Comprehensiveness Score % (IQR)	Median Reliability Score % (IQR)	Median Understandability Score % (IQR)	Median Actionability Score % (IQR)	Median Composite Score % (IQR)
Video-sharing platforms
YouTube	100 (87.5–100)	12.5 (6.3–18.0)	37.5 (32.5–43.8)	80.0 (71.1–81.8)	0 (0–16.7)	36.8 (30.7–43.0)
Facebook Watch	100 (86.3–100)	6.3 (2.3–12.5)	35.0 (35.0–45.0)	80.0 (70.0–80.5)	0 (0–41.7)	32.4 (25.9–37.3)
TikTok	100 (75.6–100)	6.3 (1.2–7.8)	35.0 (30.0–35.0)	96.9 (88.2–100)	0 (0–0)	27.5 (24.7–31.5)
p-value	0.76	0.004*, a	0.078	< 0.001*, b	0.43	0.001*, c
Level of agreement among reviewersd	W = 0.880	W = 0.976	W = 0.904	W = 0.972	W = 0.978	W = 0.970
Videos with different author qualification grading
Tier One (n = 26)	100 (94.8–100)	9.4 (2.0–13.7)	40.0 (35.0–45.0)	80.0 (71.6–80.7)	0 (0–66.7)	36.0 (31.3–39.5)
Tier Two (n = 27)	100 (89.6–100)	6.3 (3.1–13.3)	35.0 (30.0–37.5)	83.3 (62.7–96.9)	0 (0–16.7)	35.2 (29.0–42.5)
Tier Three (n = 19)	87.5 (77.5–100)	9.4 (6.3–15.6)	37.5 (27.5–40.0)	81.8 (79.5–86.4)	0 (0–0)	31.6 (25.7–36.5)
p-value	0.12	0.56	0.042*, e	0.074	0.13	0.29
“General” and “Specific” videos
General (n = 41)	100 (85.7–100)	6.3 (1.6–14.1)	35.0 (32.5–42.5)	81.8 (77.3–92.8)	0 (0–50.0)	32.1 (25.7–37.8)
Specific (n = 31)	100 (87.5–100)	9.4 (6.3–14.8)	37.5 (33.8–45.0)	77.3 (69.2–82.6)	0 (0–0)	35.1 (30.3–41.2)
p-value	0.67	0.81	0.35	0.086	0.013**	0.19

p < 0.05 based on Kruskal-Wallis test.

p < 0.05 based on Wilcoxon Rank Sum test.

Comprehensiveness scores of YouTube were significantly higher than Facebook Watch (p = 0.015) and TikTok (p = 0.004) based on Wilcoxon Rank Sum test with Bonferroni adjustment.

Understandability scores of TikTok were significantly higher than YouTube (p = 0.001) and Facebook Watch (p < 0.001) based on Wilcoxon Rank Sum test with Bonferroni adjustment.

Composite score of YouTube was significantly higher than TikTok (p = 0.001) based on Wilcoxon Rank Sum test with Bonferroni adjustment.

Inter-rater reliability based on Kendall's coefficient of concordance (p < 0.05).

No statistically significant differences found between tiers based on Wilcoxon Rank Sum test with Bonferroni adjustment.

In terms of author qualification grading of the videos, there were 26 (36.1%) Tier One, 27 (37.5%) Tier Two and 19 (26.4%) Tier Three videos. Tier One (median = 36.0%, IQR = 31.3–39.5%) and Tier Three videos (median = 31.6%, IQR = 25.7–36.5%) had the highest and lowest composite scores respectively (Table 4). Accuracy was the highest scoring quality domain (median score range = 87.5–100%), followed by understandability (median score range = 80.0–83.3%), reliability (median score range = 35.0–40.0%) and comprehensiveness (median score range = 6.3–9.4%). Tier One videos had higher reliability scores (40.0%) than Tier Two (35.0%) and Tier Three videos (37.5%). All three tiers scored a median of 0% for actionability.

There were more “general” videos (n = 41, 56.9%) than “specific” videos (n = 31, 43.1%). Overall, “specific” videos (median = 35.1%, IQR = 30.3–41.2%) had a higher composite score than “general” videos (median = 32.1%, IQR = 25.7–37.8%) (Table 4). Median accuracy scores for both “general” and “specific” videos were 100%, but median actionability scores were both 0%. “General” videos had a higher understandability score (median = 81.8%, IQR = 77.3–92.8%) compared to “specific” videos (median = 77.3%, IQR = 69.2–82.6%, p = 0.086). Conversely, “specific” videos had higher scores for reliability (median = 37.5%, IQR = 33.8–45.0%, p = 0.35) and comprehensiveness (median = 9.4%, IQR = 6.3–14.8%, p = 0.81) compared to “general” videos (reliability median = 35.0%, IQR = 32.5–42.5%; comprehensiveness median = 6.3%, IQR = 1.6–14.1%).

Discussion

This study evaluated the quality of COVID-19 vaccine videos on three top ranking video-sharing platforms – YouTube, Facebook Watch and TikTok, based on the parameters of understandability, actionability, reliability, accuracy and comprehensiveness. Overall, the quality of the videos was low, which is consistent with findings from other studies evaluating the quality of health information on video-sharing platforms.

All three platforms had low comprehensiveness scores. The videos across all the platforms covered less than 50% of the COVID-19 vaccine information in the current quality assessment tool (Appendix 2). The most comprehensive video was from YouTube called “COVID-19 Vaccine Safety: What You Need to Know (Updated)” (https://youtu.be/q-Ugzx9KuRY), which scored 48.4%. Across all three platforms, only five videos (6.9% each) provided information about suitable age groups and contraindications for the vaccines. While most of the currently approved vaccines are for individuals aged 18 and above, Comirnaty is the only vaccine that is approved for individuals 16 years and older. Incorporating this information in the videos could have increased public awareness around the eligibility to be vaccinated by this vaccine, which could potentially help increase the vaccination rates in countries that have authorised the use of Comirnaty. In addition, incorporating information about the contraindicated patient groups (e.g. allergies to the vaccine ingredients) would be important to help avoid individuals from experiencing severe or life-threatening allergic reactions, which would further assist them in making more evidence-based decisions surrounding the vaccination programmes.

Median scores for accuracy were high for YouTube, Facebook Watch and TikTok. Recently, these video-sharing platforms had implemented measures to curb the spread of misinformation. In December 2020, Facebook announced that it would begin to remove false claims related to COVID-19, including “claims about the safety, efficacy, ingredients or side effects of the vaccines”. Similarly, YouTube and TikTok also introduced measures to remove misinformation on their platforms., In addition, Tier One videos had higher accuracy scores than Tier Three videos. This trend was similar to previous studies evaluating COVID-19 pandemic information on video sharing platforms, where government agencies provided more accurate information than other sources., Interestingly, previous studies noted outlandish claims and conspiracy theories like “pharmaceutical companies have a cure but won't sell it so everyone is dying” and that the virus was linked with 5G development. However, no such claims were observed from the videos in our study. Our results showed that inaccuracies in the videos were largely due to statements not being specific. For example, one “general” video discussed that individuals under 18 years of age would not be able to receive the Comirnaty vaccine, even though this vaccine was approved for 16 years and above. Another video mentioned that pain was a side effect of COVID-19 vaccines, but was not specific about the type of pain (e.g. pain at the injection site or muscle pain).

Despite the high accuracy scores, COVID-19 vaccine information is still evolving rapidly. Since the evaluation was conducted, new side effects have been reported. For example, the European Medicines Agency found a link between Covishield/Vaxzevria and cases of blood clots, which has now been listed as a rare side effect of the vaccine. Additionally, there is new evidence surrounding the use of vaccines in adolescents, such as the Comirnaty vaccine demonstrating 100% efficacy in adolescents 12–15 years of age. As a result of the evolving COVID-19 vaccine information, the accuracy of videos will change with time. As such, users are encouraged to watch the latest videos on the video-sharing platforms, so that their chances of being misinformed are minimised.

Reliability was the third highest scoring quality domain. However, the reliability scores of all three platforms were low. Only one-third (26/72 videos, 36%) of the videos were of a Tier One author qualification grading, which meant that these videos were from authoritative sources such as the WHO, CDC, governmental authorities, and peer-reviewed journal publications. A similar proportion of videos (36%) had also provided supporting references and/or links to substantiate the COVID-19 vaccine claims made in the videos. The low number of videos that scored high in these two criteria (author qualification grading and attribution of information sources) might have contributed to the low reliability scores.

Similarly, most videos did not provide a clear statement of purpose, nor discussed commercial agenda, which could also impact the reliability scores. Among all the evaluated videos, there were five of them that had hinted a purpose for the video, even though there was no clear statement provided. For example, in the video “What's Actually in the Pfizer and Moderna Coronavirus Vaccines?” (https://youtu.be/qoorkF8FGyI), an educational component was implied in the statement made towards the end of the video, saying that “I hope learning more about these ingredients has made you feel more comfortable and empowered during your vaccine decision making” (5:51–5:57 mins). Similarly, educational components were also observed in the other four videos. Interestingly, three of the five videos were uploaded by Tier Three authors without any official expertise in healthcare/public health, which suggested that there was also an effort among the public to educate about COVID-19 vaccines. On the other hand, it was difficult to determine the presence of any commercial bias in the videos. Only the video “Doctor explains mRNA COVID-19 vaccine” (https://youtu.be/Xzlwhz0MhTw) had acknowledged that it was not sponsored by any company or organization. Currently, in most countries, people do not have a choice of which vaccine they receive, as the variety of vaccines are limited and the vaccines are given based on the supply available. In addition, certain countries also have specific policies surrounding the suitability of each vaccine. Thus, the current focus of videos could have just been to increase the awareness about COVID-19 vaccines. Concerns over commercial profiteering is one of the reasons that people are reluctant to receive vaccines. When more vaccines become available, the videos on video-sharing platforms may potentially start to introduce a commercial agenda, which may lead to an increase in vaccine hesitancy.

Actionability of the videos was assessed on three criteria – whether the material identified at least one action the viewer could take, if the video addressed the viewer when describing the actions, and if the video broke down the action into steps that the viewer could take. Although actionability scores were poor, some videos were able to advocate immediate actions to take post-vaccination. For example, the “general” videos focused on the precautions people should take and how to manage side effects post-vaccination. Some actions identified included wearing masks, social distancing and taking pain relievers or anti-inflammatory medications. In contrast, “specific” videos were less likely to discuss “actionable” information and focused more on topics such as how a particular type of COVID-19 vaccine worked, or details about vaccines like the dosing regimen and efficacy rate. A survey had found that one in five individuals were less likely to get vaccinated when they found out that they still had to adhere to preventive measures after vaccination. We postulate that “specific” videos might have omitted actions, such as masking and social distancing, on purpose as incorporating these actions might potentially increase vaccine hesitancy.

Although there was a large number of views and likes of the videos across all three platforms (median of 139,603 views and 1,264 likes), these statistics alone were not able to elucidate the overall quality of the videos. From our observation, there were no correlations between the overall quality scores of the videos and the number of likes (r = −0.044) and views (r = 0.082). There are many other reasons for “liking” a video besides for its content, for example, as a form of acknowledgement of viewing, to show support or to strengthen or maintain the relationship with the author. Some viewers may also use the “like” function as an archival tool to view the content at a later time. Furthermore, in the current climate, new COVID-19 vaccine videos are being uploaded regularly and it takes time for these videos to gather views and likes, regardless of their quality. As such, viewers are discouraged to view COVID-19 vaccine videos purely based on the number of views or likes.

In this study, it was observed that the names of some vaccines were similar. CoviVac was one of the vaccines approved for use in Russia and included in this study. However, there were two other vaccines with similar names. Covi-Vac is a vaccine developed by Codegenix in the United States and is currently in Phase 1 clinical trials. COVIVAC is another vaccine that is also in Phase 1 clinical trials but developed by the Institute of Vaccine and Medical Biologicals in Vietnam. The main differences among these vaccines are that Covi-Vac is a live-attenuated vaccine administered via the intranasal route, while CoviVac and COVIVAC are inactivated vaccines and are delivered by injection. Look-alike, sound-alike medications might lead to medication errors by healthcare professionals. Although none of the videos evaluated in this study were on CoviVac, the similarity in the vaccine names might lead to misinterpretation and confusion among users, which could potentially result in negative consequences in the healthcare setting.

Limitations and future work

A limitation of this study was that the COVID-19 vaccine information used to create the quality assessment tool only reflected the accuracy of information during the period of study. Furthermore, as this study included only videos in English, this could have limited the generalisability of the results to videos of other languages. Information for COVID-19 vaccines is constantly changing, with guidelines being updated as new evidence for the various vaccines arise. As such, the vaccine-related information in the quality assessment tool needs to be updated to assess videos in future studies.

Another limitation was the small sample sizes of videos on each video-sharing platform (35 from YouTube, 23 from Facebook Watch, and 14 from TikTok). Furthermore, only the first 20 videos from each search were selected and videos longer than 12 mins were excluded. As such, this study only captured a snapshot of information available on YouTube, Facebook and TikTok, and results for the comprehensiveness of videos might not be representative of videos with longer durations. Although Instagram ranks as the third most popular video sharing platform, it was excluded in this study because its inability to access the videos without an account might have introduced some bias based on user searches. Therefore, the results of this study might not completely represent all the COVID-19 vaccine information available. In addition, the results cannot be generalised to other video-sharing platforms, as they may contain a different set of videos and information, uploaded by different authors/sources. As content regarding COVID-19 vaccines on video-sharing platforms are constantly updated, the search results will undoubtedly change, thus future studies should consider evaluating a larger sample size of videos, videos with longer durations and from a wider variety of popular video-sharing platforms.

Lastly, in evaluating the video-sharing platforms, the understandability of the videos might be considered biased, as influenced by the healthcare professional background and education of the authors. Comprehension pertaining to the video materials as measured by the layperson was not captured in the current research. Despite this, in order to mitigate potential bias, the PEMAT tool was used because its definition of understandability had already taken into account viewers of “diverse backgrounds and varying levels of health literacy”. Nonetheless, reviewers of different socio-demographic backgrounds and health literacy should be included in future studies.

Conclusion

Many people utilise video-sharing platforms like YouTube, Facebook Watch and TikTok to obtain information about COVID-19 vaccines. However, misinformation on these platforms may contribute to vaccine hesitancy, which may impact on ongoing efforts relating to control and elimination of the disease. In this study, the overall quality of videos on these video-sharing platforms was low, but they provided accurate information on COVID-19 vaccines. Furthermore, videos from TikTok were the most understandable. Despite providing accurate information, most videos are still not comprehensive. Thus, viewers are encouraged to watch several videos before making a better-informed decision.

Financial disclosure

This research did not receive any specific grant from funding agencies in the public, commercial or not-for-profit sectors.

Contributorship

KY and LLW conceived and designed the study. RT and AP conducted the study. RT analyzed the results. KY, LLW, RT wrote the manuscript. RT and KY revised the manuscript. All authors agreed to the publication of the manuscript.

Declaration of Competing Interest

No competing financial interests exist.