YouTube as source of information on 2019 novel coronavirus outbreak: a cross sectional study of English and Mandarin content.

BACKGROUND: The current 2019 novel coronavirus outbreak is rapidly evolving. YouTube has been recognized as a popular source of information in previous disease outbreaks. We analyzed the content on YouTube about n-CoV in English and Mandarin languages.
METHODS: YouTube was searched using the terms '2019 novel coronavirus', 'Wuhan virus' and '' (Mandarin for Wuhan virus) on 1st and 2nd February 2020. First 50 videos in each group were analyzed. Videos in other languages, duplicate videos, those without an audio and duration >15 min were excluded .72 videos in English and 42 in Mandarin were reviewed. 2 reviewers classified the videos as useful, misleading or news based on pre specified criterion. Inter-observer agreement was evaluated with kappa coefficient. Modified DISCERN index for reliability and medical information and content index (MICI) score were used for content analysis.
RESULTS: These videos attracted cumulative 21,288,856 views. 67% of English and 50% Mandarin videos had useful information. The viewership of misleading Mandarin videos was higher than the useful ones. WHO accounted for only 4% of useful videos. Mean DISCERN score for reliability was 3.12/5 and 3.25/5 for English and Mandarin videos respectively. Mean cumulative MICI score of useful videos was low (6.71/25 for English and 6.28/25 for Mandarin).
CONCLUSIONS: YouTube viewership during 2019 n-CoV outbreak is higher than previous outbreaks. The medical content of videos is suboptimal International health agencies are underrepresented. Given its popularity, YouTube should be considered as important platform for information dissemination.

Introduction

In December 2019, a series of pneumonia cases of unknown cause were reported in Wuhan city in Hubei Province of China [1]. A novel strain of coronavirus was isolated from the affected patients on 7th Jan 2020 and the number of confirmed cases rapidly soared to 581 within 2 weeks [2]. The cases spread beyond Wuhan city and on 30th Jan 2020, World Health Organization (WHO) declared the 2019 novel coronavirus (2019- nCoV) outbreak as “public health emergency of international concern “(PHEIC). As per WHO situation report of 6th February 2020, 28276 cases had been confirmed globally, of these 216 were outside China in 24 countries worldwide [3].

In our current digital world, online platforms are perhaps the most accessible source of health related information for the public [4,5]. YouTube is a popular video sharing website with an estimated monthly viewership of 1 billion [6] and serves as an important source of health care related information. During the Ebola outbreak in 2014, relevant You tube videos were watched millions of times [7]. Similar viewership was reported during the 2016 Zika virus epidemic [8]. Such formidable viewership makes YouTube a double-edged sword in times of disease outbreaks. While appropriate YouTube content may benefit the government agencies and health care organizations in allaying public anxiety and enforcing measures to control the spread of disease, dissemination of wrong information can lead to public paranoia and failure to contain the infection [9].

The quality of information on YouTube has been evaluated during previous epidemics and endemics. Up to 20% of videos were reported to have misleading information about various aspects of disease during these outbreaks [8,10,11]We are not aware of any studies evaluating the role of YouTube as a source of information on 2019 -nCoV. The objectives of our study were (i) to analyze the quality of the information available on YouTube about current nCoV outbreak (ii) to compare the English language content with the information available in Chinese Mandarin henceforth mentioned as Mandarin. It is the most widely spoken language in Asia and official language of China. The current n-CoV epidemic has affected the Asia Pacific region more than the West. We think that it is therefore important to review the available content in Mandarin.

Methods

YouTube was accessed on 1st and 2nd February 2020 from Singapore using the search terms ‘2019 novel coronavirus’ and ‘Wuhan virus ‘. Search term ‘2019 novel coronavirus’ yielded 16,4000,000 results while the term ‘Wuhan virus’ showed 20,800,000 results. The search term used in Mandarin ‘武汉病毒’ is a translation for ‘Wuhan virus’ and showed 58,100,000 results. We included first 50 videos from each search term for further analysis. This screening strategy is based on previous studies indicating that the users do not go beyond the first few pages of results from a search engine [[12], [13], [14]]. The default YouTube algorithm of decreasing order of relevance was used for sorting the videos. These 150 videos were saved in a playlist for further analysis because the search results in YouTube can change on a day to day basis [7]. Uniform resource locators (URLs) of these videos were saved separately as back up. Our search methodology is in alignment with previous studies on YouTube content [[15], [16], [17]].

Of the 150 videos screened, 72 videos in English and 42 videos in Mandarin were included in further analysis (Fig. 1a, Fig. 1b A and B). Exclusion criterion were: videos in language other than English or Mandarin, absence of audio, videos lacking information on the 2019 novel coronavirus and duration more than 15 min. An analysis of 250 top YouTube channels concluded that the optimal length for a video is 10–16 min [18]. Videos with longer duration do not receive much viewership. Hence we excluded those videos from our study. The videos with subtitles were included in the group of the subtitle language. All videos were reviewed and analyzed by 2 independent reviewers (WZT and YYL). Both WZT and YYL have received formal education in English and Chinese and are fluent in both. WZT is a haematologist and YYL is a medicine trainee. Any discrepancies were resolved by a third reviewer NKB who is proficient in both English and spoken Mandarin. As this study required analysis of publicly available information, Institution Review Board approval was not required.

Fig. 1a

Screening process for English Videos.

Fig. 1b

Screening process for Mandarin Videos.

Assessment of content

Videos were classified as useful if they contained scientifically correct information about any aspect of the disease (symptoms, treatment, prevention of disease, or measures being taken to combat the disease). The videos that contained scientifically unproven (e.g., referring to nCoV as a manmade conspiracy or government propaganda) information were classified as misleading. Videos uploaded by news agencies with information on current status of the disease in terms of mortality and extent of spread only, without any information on prevention, treatment, or measures to combat the disease were classified as news update. This classification has been used in evaluation of information during previous epidemics [10,19,20].

We used modified DISCERN score to assess the reliability of the videos [15]. This score allocates one point each for clarity, reliable source of information, lack of bias, reference supplementation and mention of uncertainty (Table 1 ). A maximum score of 5 indicates highest reliability for a particular video. For evaluating the content, we used medical information and content index (MICI) as previously defined by Nagpal et al. during the Ebola epidemic [7]. This scale uses a 5 point Likert scale to assess five components of medical information included in the videos: prevalence, transmission, clinical symptoms, screening/testing, and treatment/outcomes of the disease. The study team devised set of criterion to be used for grading each component of MICI score on a scale of 1–5. For example a video would be scored 5 for the treatment/outcome component if it mentions all of the following: the symptoms can be self-resolving but some cases can become dangerously ill requiring hospitalization or intensive care unit, disease can lead to death, treatment is mainly supportive, vaccinations not yet available. The full list of criterion is available as Supplementary Table 1. The kappa coefficient of agreement was used to determine degree of agreement between the two researchers.

Table 1

Modified DISCERN criterion.

1.Are the aims clear and achieved ?

2. Are reliable sources of information used? (i.e., publication cited, speaker is a certified physician)

3. Is the information presented balanced and unbiased?

4. Are additional sources of information listed for patient reference ?

5. Are areas of uncertainty mentioned?

From Radonjic et al. [15]

Source of videos was categorized into one of five groups: academic institutions or hospitals, news agencies, government agencies like regional health departments, WHO or independent users.

The authors are aware of the uncertainties regarding this disease. For example, there is still no consensus on the source of disease outbreak or the transmissibility of the virus, exact incubation period of the virus, and the proportion of infected people who develop disease. Before grading the videos, available scientific literature [[21], [22], [23]] and relevant information on WHO website was reviewed.

Analysis

Version R 3.5.3 was used for analysis. Unpaired Student t-test was used to test the hypothesis of difference in means for all variables. A p-value of 0.05 was considered to be significant. Given the small sample size, univariate logistic regression was used to find the odds of posting a useful video based on source.

Results

72 videos in English and 42 videos in Mandarin were included for analysis after applying exclusion criterion (Fig. 1a, Fig. 1b). The total duration of all videos was 8.45 h. These 114 videos attracted a cumulative number of 21,288,856 views and 259108 likes.

The kappa coefficient of agreement regarding the usefulness of the videos was 0.72 (P value < 0.001). 60% videos were classified as useful (n = 68), 8% misleading (n = 9) and 32% as news (n = 37). Baseline characteristics of the videos are summarised in Table 2 .

Table 2

Baseline characteristics of videos included for analysis after applying exclusion criterion.

Video characteristics	Videos in English (n = 72)		P value*	Videos in Mandarin (n = 42)		P value*
	Useful (n = 47)	Misleading (n = 2)		Useful (n = 20)	Misleading (n = 7)
Mean number of views	288545.30	1621.5	0.00001	91949.90	151868.70	0.30
Mean number of views/day	99083.53	438.33	0.0002	21739.39	73222.72	0.14
Mean number of likes	3889.27	9.50	0.08	640.40	1037	0.55
Mean number of likes/day	1118.37	2.58	0.07	207.57	473.57	0.35
Mean number of days on YT	6.42	3.50	0.02	8.25	3	0.0006
Mean length of videos (mins)	4.21	4.22	0.99	3.30	7.20	0.10
Modified DISCERN score	3.12	0	NA¶	3.25	0	NA¶

*p values calculated for difference in characteristics between useful and misleading videos for each group.

¶p value was not calculated as misleading videos were not scored using the DISCERN index.

The majority of videos were uploaded by news agencies (76%, n = 86) followed by independent users (11%, n = 12). WHO contributed 3 videos (Table 4 ). One of these videos was in English language with Mandarin subtitles and had 29466 views. There were no videos by CDC. No videos were shared by academic institutions in Mandarin. Independent users were 7 times less likely to upload useful information than academic institutions (Table 5 ).

Table 4

Source wise distribution of different video types.

		English (n = 72)			Mandarin (n = 42)
Video source	Total videos (n = 114)	Useful (n = 47)	Misleading (n = 2)	News (n = 23)	Useful (n = 21)	Misleading (n = 7)	News (n = 14)
News agencies	87	27	1	22	19	4	14
Academic institutions/hospitals	11	11	0	0	0	0	0
WHO	3	2	0	0	1	0	0
Regional health departments	1	1	0	0	0	0	0
Independent users	12	6	1	1	1	3	0

Table 5

Likelihood of videos being useful according to source.

Source of video	Odds ratioa	95% CI	p-value
Academic institutions/hospitals/WHO	1.00
Independent users	0.13	0.006–1.04	0.085
News agency	0.85	0.042–6.401	0.894

Univariate logistic regression analysis was used to calculate odds ratio.

Despite being of similar length, misleading videos in English attracted significantly lesser number of views and likes than useful videos. Conversely, misleading videos in Mandarin were of longer duration than useful videos and attracted more number of views. This difference was however not statistically significant.The statements made in misleading videos are available as Supplementary Table 2.

More than 70% of useful videos in English contained information about the prevalence, transmission and treatment/outcomes of nCoV disease. Almost half of the useful videos did not address transmission or clinical symptoms (Table 3). The mean MICI scores for each component were low. The majority of mandarin videos focussed on transmission similar to their English counterparts. 90% of useful Mandarin videos relayed no information on screening or testing of nCoV disease.

Table 3

Detailed content analysis of useful videos based on MICI scores.

	English (n = 47)		Mandarin (n = 21)
Component of MICI scale evaluated	No. (%) of videos with information	MICI scoreMean (SD)	No. (%) of videos with information	MICI scoreMean (SD)
Prevalence	33 (70.21%)	1.47 (1.3)	6 (28.6%)	0.63 (1.01)
Transmission	43 (91.49%)	2.0 (1.28)	17 (81%)	2.18 (1.55)
Clinical symptoms	24 (51.06%)	1.06 (1.36)	15 (71.4%)	2.28 (1.86)
Screening/testing	25 (53.19%)	0.66 (0.95)	2 (10%)	0.10 (0.31)
Treatment/outcomes	35 (74.47%)	1.06 (1.04)	11 (52.4%)	1.1 (1.28)
Total MICI scoreMean (SD)		6.71 (3.25)		6.28 (3.72)

Discussion

YouTube viewership of content related to the 2019 n-CoV epidemic appears higher than previous disease outbreaks [10,20]. This is evident by the much higher number of mean views/day of the useful videos in our study. This can be partly explained by the fact that we performed this analysis early in the course of disease evolution. The mean number of days since upload on YouTube for useful videos was 6 in our study compared to 61 days as reported by Pathak et al. during the 2014 Ebola virus disease outbreak. It has been reported that the viewership of videos tends to decrease over time and that could lead to higher number of views/day in the initial days of video upload. Regardless, the difference is substantial and high viewership of content in our study should be considered as a reflection of growing popularity of YouTube for health related information. The higher number of views can also be explained by the global spread of this disease compared to previous outbreaks like Ebola which was mainly restricted to Africa.

The current content on YouTube in English is mostly informative. There were only 2 misleading videos amongst the included 72 videos. These results are reassuring. Previous studies have reported that about 15–30% of information on YouTube about a disease can be misleading [[24], [25], [26]]. The lower percentage of misleading videos in our study can be explained by the difference in source of these videos compared to previous studies. Individual users accounted for only 11% of total videos while news agencies contributed 68%. This is not surprising given the global nature and rapid spread of this outbreak. This has attracted attention of news agencies worldwide. Videos uploaded by independent users are more likely to contain misleading information than government/news agency videos [8].

The modified DISCERN score of videos which were characterized as useful is of concern too. There was only 1 video with a perfect DISCERN score of 5 indicating highest reliability. 20% of the useful videos had DISCERN score of less than 3 indicating low reliability. Most of the videos did not mention areas of uncertainty or additional sources of patient information. International health agencies like WHO only had 2 videos while CDC had none in the top 100 English videos related to this disease. Under representation of international health agencies on YouTube has previously been reported [8].

Previous studies have focussed on analysis of YouTube videos only in English. YouTube is gaining popularity for its content in non-English languages too. In the first week of 2019, a total of 243,254 videos were posted on YouTube, out of these 67% were in languages other than English [27]. This prompted us to evaluate the information available on YouTube about nCoV in Mandarin and compare it to the content in English. The mean DISCERN scores were similar in 2 groups indicating comparable reliability. Similarly, comparable but very low mean total MICI scores were noted in both groups, suggesting overall paucity of good quality scientific information on n-CoV outbreak. The information pertaining to screening and testing was worst with mean MICI scores of 0.95 and 0.31 in English and Mandarin groups respectively. Under representation of government agencies was noted in Mandarin videos as well.

News updates constituted most of the Mandarin videos too, however compared to English new agencies, the narrators of news clips in Mandarin often commented on how Chinese government could have acted differently to contain this epidemic early. No videos were shared by academic institutions and hospitals in Mandarin which contributed to slightly higher percentage of misleading videos in Mandarin.

Contrary to the English videos, the misleading videos in Mandarin were more popular than useful videos, as evidenced by higher mean number of likes. The misleading information in native language at epicentre of outbreak might have led to undue public distress during early phase of outbreak. YouTube thus should play a more active role in screening the content of such videos and censoring misleading information.

Limitations

Our study has many limitations. This study presents only a cross sectional snapshot of the available information. The viewership and content on YouTube change on a daily basis. We are unable to comment on how the viewership and content regarding 2019 nCoV will change as the outbreak evolves. We did not analyze other video sharing platforms such as health sharing websites or news channel websites. In addition, YouTube is not accessible from China. Instead, a platform called Youku is used for video sharing. This affects the applicability of results in China which is the epicentre of this disease outbreak. Restriction of YouTube in China also explains the relative lesser number of views of Mandarin language videos compared to those in English. Nonetheless, Mandarin remains the widest spoken language in Asia Pacific region and videos in this language serve as an important source of information to native Mandarin speakers in this region. Another limitation of our study is the possibility of inter and intra observer bias though we used the kappa coefficient to ensure that the results were not too different.

Our search was limited to first 50 videos for each keyword. It has previously been explained in multiple studies that most Internet users do not look beyond the first 50 search results [16,28,29]For the search of videos in Mandarin, we used the Mandarin translation of ‘Wuhan virus’ and not ‘2019 Novel Coronavirus’. This strategy was based on our preliminary review of number of videos with each search term. The Mandarin term for 2019 novel coronavirus ‘2019 新型冠状病毒’ showed only 380000 videos compared to 58100000 videos identified with the term ‘武汉病毒’ which means Wuhan virus in Mandarin. This indicated that the former term was not very popular in Mandarin language and hence was omitted from further inclusion. YouTube's search can be affected by the previous search history. We rectified this issue by using a cache cleared browser and creating a fresh account for the search.

Conclusion

In conclusion, the available information on YouTube about 2019 n-CoV outbreak attracted high viewership. This clearly establishes YouTube as a popular platform for information seeking in the current n-CoV epidemic. As the current outbreak evolves, international health agencies and academic institutions should release more videos with information about screening and testing of this disease. YouTube should consider screening and removing videos with misleading information to prevent public paranoia in current state of emergency.

Funding

No funding was required to conduct this study.

CRediT authorship contribution statement

Priyanka Khatri: Conceptualization, Data curation, Methodology, Project administration, Writing - original draft, Writing - review & editing. Shweta R Singh: Data curation, Formal analysis, Writing - review & editing. Neeta Kesu Belani: Writing - original draft, Writing - review & editing. Yin Leng Yeong: Data curation, Writing - original draft, Writing - review & editing. Rahul Lohan: Conceptualization, Methodology, Writing - original draft, Writing - review & editing. Yee Wei Lim: Conceptualization, Methodology, Supervision, Writing - review & editing. Winnie ZY Teo: Conceptualization, Methodology, Project administration, Data curation, Writing - original draft, Writing - review & editing.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.