YouTube as a source of information on COVID-19 vaccination in rheumatic diseases.

As the most well-known and popular video-sharing platform around the world, YouTube is an influential tool for the dissemination of health-related information. In addition, considering the increase in obtaining information from internet-based sources in pandemic conditions, YouTube has become more important in the presentation of information related to COVID-19. Therefore, the aim of this study was to evaluate videos related to COVID-19 vaccination in rheumatic diseases (RD) on YouTube. In this descriptive study, 334 video URLs listed with six search terms were recorded (26 July 2021). Three quality groups (high, intermediate, and low) were created based on the Global Quality Scores (GQS). Video sources were identified and various video parameters were compared between the quality groups. Following the implementation of the exclusion criteria, 56 videos remained for further analysis; of which 37 (66.07%) were evaluated as high quality, 12 (21.42%) as intermediate quality, and 7 (12.51%) as low quality. No significant difference was determined between the quality groups in per day values of views, likes, dislikes, and comments. The sources of high-quality videos were pharmaceutical company (n = 1; 100%), pharmacist (n = 1; 100%), society-organization (n = 17; 85%), and academic (n = 3; 75%). Although two-thirds of the videos were high quality, it should be kept in mind that intermediate and low-quality videos are also available. Users should not assume the quality of the videos based on the number of views, likes, dislikes, and comments, but should focus more on video sources.

Introduction

The immune system alterations that emerge in the course of various rheumatic diseases (RD) create a potential that cannot be ignored for the occurrence of intense viremia and the severe new type of coronavirus disease 2019 (COVID-19) [1]. High disease activity, comorbid disorders secondary to RD, and medications are other potential risk factors for COVID-19 in this group of patients [2, 3]. The COVID-19 vaccines, which have recently become widespread all over the world, have revealed a tremendous expectation that they will play a role in ending the pandemic, which has deeply affected the whole of society [4].

Although there are recommendations in the guidelines to prioritize vaccination of RD patients, the fact that patients with RD and those using immunosuppressive drugs were excluded from the clinical trials of COVID-19 vaccines may cause vaccine hesitation [5, 6]. It is obvious that this situation motivates RD patients to reach accurate and reliable information. Accessing information on the Internet is one of the important sources in this regard. A considerable portion of Internet consumers has the potential to use YouTube, the second most preferred website globally after Google, to provide more and detailed information [7]. YouTube, the most popular video-sharing platform, provides a wide range of content to users, with more than 500 h of video uploaded per minute [8]. While it has the advantages of being free, easy to access, and an extensive information network, there is concern that it will remain a manipulated target for presenting incorrect information about COVID-19 vaccines [9]. In addition, videos that present misinformation about COVID-19 vaccines can pose a significant barrier to adherence to vaccination programs, particularly among those at risk of severe disease.

There have been many studies evaluating COVID-19-related YouTube videos from various perspectives [7, 10–12]. However, there are no studies analyzing videos on COVID-19 vaccination in the specific group of RD patients. Therefore, the primary aim of this study was to determine the quality and reliability of COVID-19 vaccination videos related to RD. It was also aimed to reveal the sources of high-quality videos, and if available, anti-vaccine videos. The third aim was to compare various parameters between high-, intermediate- and low-quality videos.

Materials and methods

This descriptive study was conducted on YouTube (http://www.youtube.com) with the search terms ‘COVID-19 vaccination rheumatic disease’, ‘COVID-19 vaccine rheumatic disease’, ‘SARS-CoV-2 vaccination rheumatic disease’, ‘SARS-CoV-2 vaccine rheumatic disease’, ‘coronavirus vaccination rheumatic disease’ and ‘coronavirus vaccine rheumatic disease’ (26 July 2021). To list the videos in the most comprehensive way, two authors (BFK and AA) came together and determined these six search terms by consensus. Previous research has revealed that a significant proportion of Internet consumers only examine the first three pages of search results (a total of 60 videos from 20 videos on one page) [8, 13]. Although YouTube has switched to continuous listing instead of listing pages, it was planned to include the first 60 videos for each search term, based on similar studies [14, 15]. Since 34 videos were listed for the search term ‘SARS-CoV-2 vaccine rheumatic disease’, the targeted 60 videos could not be reached. Video listing was provided based on the number of views. Thus, it was aimed to evaluate the videos that have the greatest impact potential on society. Access was performed with the incognito form of Google Chrome to prevent bias based on search history and cookies. A total of 334 video URLs were recorded and the review process was maintained by the two researchers (BFK and AA) blinded to each other’s evaluation results. Only English videos were included in the evaluation process. Videos not relevant to the topic, videos in languages other than English, repeated videos, and videos with audio–video problems preventing accurate assessment were excluded. If an author had doubts about whether to include a video or not, the two authors (BFK and AA) met and made the final decision by consensus.

Assessment of quality

The video contents were reviewed in terms of vaccine development processes, COVID-19 vaccine types, main features and differences of vaccine types, clinical efficiency of COVID-19 vaccines, safety of COVID-19 vaccines, dosage regimen, administration techniques, potential interactions with antirheumatic drugs, adverse effects, and cautions. The authors prioritized the following questions while evaluating the videos: is COVID-19 vaccination described in all aspects? Is the information provided in accordance with scientific data and up-to-date references? Is the information supported by accurate sources? Is the information presented without bias and in a balanced way that covers all aspects of the issue? Is the information proper and helpful for patients? Is the information and data up-to-date? The pointers for high-quality videos are summarized in Table 1.

Table 1

The pointers for high-quality videos

Explaining the effects of COVID-19 clearly

Presenting COVID-19 vaccine types, main features, and differences in vaccine types

Describing the clinical efficacy of COVID-19 vaccines with appropriate references

Giving information about the safety, dosage regimen, administration techniques of COVID-19 vaccines

Addressing possible interactions of COVID-19 vaccines with antirheumatic drugs

Explaining the effect of COVID-19 vaccines on rheumatic disease activity

Presenting side effects and cautions associated with COVID-19 vaccines in rheumatic diseases

The two authors (BFK and AA) considered the aforementioned content parameters/questions and independently rated each video using the Global Quality Scale (GQS). Thus, the authors were prevented from being influenced by each other while scoring. The GQS was mainly developed for rating internet-based resources [16]. Scoring ranges from 1 to 5 points, with higher scores indicating better quality. Based on the GQS scores, the videos were divided into three quality groups. Videos with a score of 4 and 5 were classified as high quality, videos with a score of 3 were classified as intermediate quality, and videos with a score of 1 and 2 were classified as low quality [14, 17]. Where the scores of the two authors did not match, a third researcher (ARS) conducted the assessment and his scoring was accepted as the final result.

Assessment of reliability

Reliability evaluations were performed using the modified DISCERN tool which has five yes–no questions. This is a shortened version of the original form. This tool allows the videos to be assessed in terms of clarity—clearness, understandability, information resources, bias, balanced information presentation, and addressing unclear-controversial issues. A response of “yes” is scored as one point and “no” as 0 points to give a total score between 0 and 5 points. Higher scores indicate greater reliability [18, 19].

Video parameters

The total duration of the videos was recorded in seconds. The upload date of the videos was noted and the period between this date and July 26, 2021 was determined as days. In addition, the total and per day views, likes, dislikes, and comments data were recorded.

Video sources

Video sources were categorized as follows: society-organization, physician, pharmacist, health-related website, academic, independent user, patient, non-physician health personnel, pharmaceutical company, and news.

Statistical analysis

All the data were analyzed statistically using the Statistical Package for the Social Sciences version 20.0 software (SPSS Inc., Chicago, IL, USA). Prior to the analyses, conformity to normal distribution was checked with the Shapiro–Wilk test. Data were reported in the text, tables and figures as median (minimum–maximum) values, number (n), and percentage (%).Three quality groups were formed and comparisons between these groups were made with the Kruskal–Wallis test. The Kappa coefficient was calculated to assess the consistency between the video scorings of the authors (BFK and AA). Interpretations on statistical significance were made based on a value of 0.05.

Results

Initially, a total of 334 videos were listed and duplicate videos were identified. Only one of the repeated videos was evaluated in the further analysis and 182 videos were excluded. The remaining 152 videos were reviewed for other exclusion criteria, and 69 irrelevant and 27 non-English videos were removed. Then, assessments were performed on a total of 56 videos that met the determined criteria (Fig. 1). The distribution of the videos according to the sources is as follows: Society-organization (n = 20; 35.71%), physician (n = 15; 26.78%), health-related website (n = 11; 19.64%), academic (n = 4; 7.14%), patient (n = 2; 3.57%), pharmaceutical company (n = 1; 1.78%), news (n = 2; 3.57%), and pharmacist (n = 1; 1.78%) (Fig. 2). There was no video from any independent user and non-physician health personnel. The general characteristics of the videos provided by each source are presented in Table 2.

Fig. 1

Flowchart revealing the selection of YouTube videos

Fig. 2

Categorization of the videos according to sources. a Society-organization; b physician; c health-related website; d academic; e patient; f pharmaceutical company; g news; h: pharmacist

Table 2

Parameters of the videos according to the sources

Source	Durationa	Number of viewsa	Number of likesa	Number of dislikesa		Number of commentsa
Society-organization	3597 (739–64,919)	1398.50 (200–9035)	23 (3–81)	1.50 (0–12)	0 (0–42)
Physician	507 (56–4692)	2563 (32–51,018)	32 (1–375)	1 (0–57)	17 (0–514)
Health-related website	661 (59–2511)	1356 (59–17,864)	15 (0–136)	1 (0–20)	2 (0–61)
Academic	3889 (546–4923)	716 (267–1216)	12 (1–30)	1 (0–3)	1 (0–4)
Patient	1003.50 (968–1039)	14,589 (4948–24,230)	252.50 (104–401)	12 (4–20)	151.50 (61–242)
Pharmaceutical companyb	3525	104	1	0	0
News	122.50 (112–133)	5526 (583–10,469)	30 (2–58)	8 (0–16)	39 (2–76)
Pharmacistb	1650	3877	228	8	70
Generalc	1038.50 (56–6491)	1409 (32–51,018)	23 (0–401)	1 (0–57)	2 (0–514)

aAll data are expressed as median (minimum–maximum)

bOne video was available from this source

cGeneral features of the videos are presented

The GQS-based quality groups were created and the videos were classified into one of three quality groups according to the final scores. Of the total 56 videos, 37 (66.07%) were classified as high quality, 12 (21.42%) as intermediate quality, and 7 (12.51%) as low quality. The sources of the high- and low-quality videos were evaluated separately. Sources providing high-quality videos with the highest percentage were pharmaceutical company (n = 1; 100%), pharmacist (n = 1; 100%), society-organization (n = 17; 85%), and academic (n = 3; 75%). At the other end of the quality spectrum, the source that provided low-quality videos at a high rate was news (n = 2; 100%) (Fig. 2). In addition, no anti-vaccine video was detected.

The video parameters were compared between the quality groups and per day values were used as a basis for more accurate results. No significant difference was detected between the three quality groups in respect of the per day values of views, likes, dislikes, and comments data (p > 0.05). A significant difference was determined between the groups in the modified DISCERN score (p < 0.001) (Table 3).

Table 3

Comparison of the video parameters between the low-, intermediate- and high-quality groups

Video quality	DISCERN scorea	Views per dayb	Likes per dayb	Dislikes per dayb	Comments per dayb
Low*	3 (2–3)	13.28 (1.72–97.08)	0.12 (0–0.73)	0 (0–0.10)	0.04 (0–0.36)
Intermediate*	4 (3–4)	5.18 (0.95–248.64)	0.07 (0–0.88)	0 (0–0.06)	0 (0–3.29)
High*	4 (4–5)	14.79 (1.52–219.90)	0.25 (0–3.01)	0.01 (0–0.27)	0.01 (0–1.81)

*All data are expressed as median (minimum – maximum);ap < 0.001; bp > 0.05

The Kappa value was calculated as 0.82.

Discussion

This study was conducted to evaluate the acquisition of internet-based information, which has become more important during the COVID-19 pandemic. As one of the platforms that dominate this field, it was preferred to conduct the study on YouTube. Although there is a previous YouTube study on COVID-19 vaccination, no research has been identified on COVID-19 vaccination in RD [7]. Because of the basic characteristics of their diseases and various immunosuppressive drugs they use, patients with RD should be considered separately from the general population. Priori et al. [20] revealed that COVID-19 vaccine hesitancy is higher in patients with RD compared to healthy controls. They reported the reasons for the hesitancy as fear of adverse events related to disease, fear of disease worsening, fear of adverse events regardless of the disease, and distrust in COVID-19 vaccines. In view of this knowledge, we considered that a YouTube study on COVID-19 vaccination in RD would be more beneficial. This study, on the particular issue of COVID-19 vaccination in RD, revealed that two-thirds of the videos were of high quality and relatively few low-quality videos were presented. In addition, high- and low-quality video sources were presented. Considering the difficulties of hospital admissions and obtaining physician opinions in pandemic conditions, Internet-based platforms have the potential to disseminate accurate high quality, or incorrect–misleading information. This suggests that the results of the current study should be carefully evaluated.

The results of this study revealed that 66.07% of the videos were of high quality and 12.51% were of low quality. In contrast, COVID-19-related YouTube studies conducted between March and June 2020 reported substantially higher rates of videos presenting incorrect or misleading information [10, 21]. The difference in results is most likely due to the COVID-19 medical misinformation policy that YouTube announced in October 2020 [22]. YouTube does not allow videos related to COVID-19 that have the potential for serious harm and unfavorable risk to be made available on the platform. Under this policy, videos with content such as ‘COVID-19 vaccines cause death’, ‘COVID-19 vaccines are used to reduce population’, ‘COVID-19 vaccines are ineffective in stopping the pandemic’ and ‘COVID-19 vaccines contain microchips’ are blocked by YouTube. As a result of this policy, anti-vaccine video was not detected in our study. The handicaps of this system are the difficulty of detecting more insidious forms of misinformation and the lack of consensus on some issues [23]. When the results of this study are compared with those of previous COVID-19 studies, it can be accepted that this policy is partially functional. However, this policy should not be completely relied upon and it should be kept in mind that incorrect or misleading information may still be presented on YouTube.

The three sources that produced the highest number of videos in this field were society-organization, physician, and health-related website, respectively. Sources of high-quality videos were pharmaceutical company, pharmacist, society-organization, and academic, while news provided a high percentage of low-quality videos. In another YouTube study on the COVID-19 and rheumatic disease link, high-quality video sources were listed as academics, physicians, and pharmaceutical companies [10]. Consistent with the current study results, the news as a non-factual source of information was predominant in a YouTube study, in which videos related to COVID-19 were evaluated [24]. The current study results suggest that while high-quality video sources produce videos in the light of scientific data, news may present incorrect–misleading information to attract users’ attention and increase the number of interactions. Low-quality videos provide content that is far from the scientific approach and aims more views and attention. They do not support the data and information with appropriate references and personal opinions take precedence. Low-quality videos do not progress systematically and do not cover all aspects of the subject. Therefore, internet users cannot find answers to most of the questions in their minds. Furthermore, the inaccurate information they present might lead to misunderstanding and even incorrect behavior. Users should attach importance to checking the source and should prioritize high-quality video sources when obtaining health-related information from YouTube. Awareness should be increased for video production that provides high-quality information among pharmaceutical companies, pharmacists, associations–organizations, and academicians, and these sources should be actively involved in Internet-based platforms and provide accurate information and up-to-date content.

Social media platforms have played a substantial role in the COVID-19 pandemic course, and the flood of pandemic-related content has been rubbed ‘infodemic’ [25]. Although social media platforms have numerous benefits, it should be noted that there may be drawbacks linked with misinformation, unethical promotion, and unprofessional behavior [26]. The spread of incorrect and misleading information regarding COVID-19 can have unintended consequences. It can distract individuals from appropriate actions that help protect their own and public health, as well as to contribute to the emergence of additional hazardous behaviors that let COVID-19 spread faster [27]. In addition, the dissemination of incorrect and misleading information about the COVID-19 vaccination in rheumatic diseases on the Internet may increase COVID-19-related morbidity and mortality in these patients who are in the risk group.

YouTube is an interactive platform. Users can share positive or negative comments under the video. Regardless of this, they can provide quick feedback by simply clicking ‘like’ or ‘dislike’. The data were extracted from YouTube covering the number of views, likes, dislikes, and comments for each video. Then, the per day values of these parameters were calculated. These data were compared between the quality groups and no significant difference was detected in the per day values of views, likes, dislikes, and comments. A significant difference was found in the modified DISCERN score comparison and the highest values were in the high-quality group. However, these results do not confirm that internet users tend to view high-quality videos in this area. Although more than half of the videos were of high quality, it can be considered as a handicap that there is no difference between the quality groups in terms of the parameters specified. Internet users may have difficulty distinguishing and selecting high-quality videos, which can result in the spread of incorrect and misleading information. Another interpretation that can be drawn from these results is that the data of views, likes, dislikes, and comments per day should not be considered as an indicator of quality. It should be noted that these data can be manipulated and data can be changed with various links to YouTube videos. For all these reasons, users should not attach great importance to the specified video parameters and should focus more on video sources.

This study included the general limitations of YouTube studies. First is that as in all studies carried out with this methodology, only a snapshot of a certain moment is taken. YouTube has a dynamic structure and content that is constantly changing and expanding. Results may vary in a similar study conducted at a different time. Another handicap was that only English videos were evaluated. Although a sufficient Kappa score was obtained as an indicator of inter-rater agreement, video evaluations are subjective. Finally, as duplicate, non-English, and irrelevant videos were excluded, a relatively low number of videos was obtained despite the use of six search terms.

Conclusion

The fact that two-thirds of the videos were of high quality and the rate of low-quality videos was approximately 10% suggests that the YouTube policy to prevent the spread of COVID-19-related misleading information is partially beneficial. It should be kept in mind that despite YouTube blocking, misleading information can still be presented on the platform. YouTube should be thought of as a mixed platform where useful and accurate information can be presented as well as misleading and incorrect information. Therefore, its impact on individuals is associated with orientation toward the right videos and sources. Although two-thirds of the videos are in the high-quality group, users can still access false and misleading information. Nevertheless, given the small number of low-quality videos, the overall influence of YouTube videos on COVID-19 vaccination in patients with RD can be positive. Patients and the public in general should be made aware that to be able to obtain accurate and high-quality information, video sources should be focused on instead of parameters such as numbers of views, likes, dislikes, and comments. High-quality video sources, such as pharmaceutical companies, pharmacists, societies-organizations, and academics should be encouraged and motivated to produce more online content.