Assessing information on YouTube™ as a quality source for the treatment of varicoceles.

INTRODUCTION: YouTube™ has grown into one of the largest disseminators of health care information. We assessed the quality of information on varicoceles and their treatment, available on YouTube™.
METHODS: Using a YouTube™ search query with the keyword "varicocele," the quality of the first 50, nonrepeat videos in English were assessed as a representative group for the topic. DISCERN and Patient Education Materials Assessment Tool for Audiovisual Materials (PEMAT-AV) standardized tools were utilized by three independent reviewers to grade the quality of these videos based on content, understandability, and actionability.
RESULTS: The average and median DISCERN score was 31.34 (±9.37) and 31 (interquartile range 25-35), respectively, indicating poor quality. The interrater reliability (IRR) scores ranged from 0.51 to 0.93, indicating fair to excellent reliability. The average PEMAT-AV understandability and actionability scores were 69.8% ±15.4% and 11.0% ±24.6%, respectively, indicating mostly understandable but poor actionability. The t-test results showed that international videos scored higher without statistical significance in the DISCERN or PEMAT-AV scores (P = 0.18, 0.59, and 0.20).
CONCLUSIONS: The current quality of videos on YouTube™ on the topic of varicoceles is of poor quality due to a lack of a holistic approach in explaining the wide range of treatment options available. With the ease of access to produce and disseminate health information, there is a need to create high-quality videos on varicoceles that empower a patient to make an informed decision. Copyright:

INTRODUCTION

With the explosion of online content, it is now common for physicians to encounter patients who have performed their own research on the internet to determine the treatment option for their perceived need.[123] This ease of access to information has been shown to be beneficial to the patient-physician relationship; however, this benefit is realized only if the information is shared with the physician and a further discussion is had about how best to proceed.[4] However, this largely relies on the assumption that patients are accessing quality information related to their condition, an area in which YouTube™ is minimally regulated. Studies analyzing the quality of videos on YouTube™ of other common health conditions have shown quality of information is lacking.[56789] Urological diseases and procedures addressed on YouTube™ are not immune to these issues as a study found that there is a negative correlation with scientific quality and viewer engagement.[9] This can be alarming as the percentage of urologist using YouTube™ and other social media platforms is increasing.[10]

Research has already been conducted on assessing information on urological conditions;[111213] however, information related to varicoceles has yet to be addressed. Varicocele is prevalent in up to 15% of healthy men and up to 35% of men with primary infertility.[14] Varicocele is widely accepted to be the most common identifiable cause of male infertility through a mechanism that is not completely understood.[15] Not all patients who are diagnosed with varicocele require treatment, placing more onus on the patient to choose whether or not to treat. In particular, patients who are trying to address issues of fertility or with mild scrotal pain will most likely try to supplement their knowledge before approaching more invasive treatments. The aim of this study is to the examine the quality, understandability, and accessibility of varicocele-related videos on YouTube™ from the patient viewpoint to better understand how this information could affect patient decision-making.

METHODS

Video selection requirements

A YouTube™ video search was conducted on October 1st, 2020, using the keyword “varicocele.” This search was conducted under an incognito webpage to limit YouTube's optimize search that is specific to a reviewer's past video history. Since the IP address used to access the YouTube™ server was in the United States, the search was used to represent a standard United States user. Previous research has shown that 41% of viewers click on entries on the first page, with only 20% searching beyond the third page.[1617] However, YouTube™ has updated its search layout by continuously adding new videos as a user reaches the bottom of the page, dubbed the “infinite scroll.” Therefore, the first 50 videos, representing 2.5 pages of videos on the older layout (20 videos per page × 2.5 pages = 50 videos) that fulfilled any of the following requirements had its title and URL recorded:

General description of varicoceles

Pathophysiology of varicoceles

Diagnosis of varicoceles

Treatment options of varicoceles

Complications of varicoceles

Personal experiences with varicoceles or treatment of varicoceles.

Videos were excluded if they were duplicates, not in English, or if the primary goal was unrelated but mentioned varicocele (e.g., videos about vasectomy or female infertility). Other video data including source category (e.g., private practice, vlogs), video category (e.g., Educational, personal experience), upload date, duration, likes, and dislikes were also recorded.

Video review

Two medical students and one senior urological resident examined the included videos utilizing two tools: The DISCERN tool to assess quality and the Patient Education Materials Assessment Tool for Audiovisual Materials (PEMAT-A/V) for understandability and accessibility.

DISCERN is a standardized survey usable by both untrained laypersons and health professionals that is typically used for the assessment of written information on treatment choices for a health problem,[1819] which was modified to reflect the media used in this study. The 16-question survey has 8 questions that address the reliability of the source, 7 questions that assess the quality of the treatment options, and 1 question regarding the rater's evaluation of the entire publication. Each question is scored between 1 and 5, with 5 defined as a definite “yes” in accomplishing the goal of the question and 1 as a definite “no.” Partial credit of a 2–4 is given if it meets criterion to some extent. The score from each category, except the rater's evaluation, is summed (total of 15 categories), giving a score between 15 and 75. Scores between 63 and 75 are considered as “excellent,” 51–62 as “good,” 39–50 as “fair,” 28–38 as “poor,” and <27 as “very poor.” Summary statistics for each question were calculated and sorted to assess the weaknesses and strengths of the current information on varicoceles.

PEMAT-A/V is another standardized survey to analyze audiovisual media on the understandability and actionability of the information provided.[20] The PEMAT-AV contains 17 criteria, with the first 13 assessing understandability and the last 4 assessing actionability. One of the PEMAT-A/V criteria was completed excluded due to media being assessed are videos, with others excluded in some videos as they did not apply. These criteria are scored as either agree or disagree using a 1 or 0, respectively, without partial credit. Summary statistics were also calculated and sorted to assess the weaknesses and strengths of the current information on varicoceles.

In addition, a large number of the videos were found to be produced by authors outside the United States. The United States versus World production analysis was conducted using a homoscedastic t-test to compare the means of the two groups for the DISCERN, PEMAT-AV understandability, and PEMAT-AV actionability scores.

Statistical analysis

Statistical analyses were perform using Microsoft Excel (Microsoft Excel 16.0, Microsoft, Redmond, WA, USA) and Google Sheets (Google LLC, Mountain View, California, USA). An IRR was calculated to validate the DISCERN and PEMAT-AV scores between the three raters. Linear regressions between run time to mean DISCERN scores and number of views to mean DISCERN scores were created. A one-sample hypothesis testing for correlation was conducted on both linear regressions.

RESULTS

For general characteristics of the 50 videos, the average length of each video was 6 min and 17 s with an average of 100,546 views, 460 likes, 44 dislikes and 3 years and 5-month-old. Fifty-six percent of the videos featured a US licensed physician. Of note, 24% of the videos featured non-US licensed physicians. Only 4% of the videos contained false statements, all of which came from the same channel that promoted nonscientific-based claims of natural resolving varicoceles such as “a proper diet will help minimize the (renal) nutcracker syndrome.”[21]

DISCERN analysis

The descriptions and average with standard deviation of each question are found on Table 1. The videos achieved a median DISCERN score of 31 (interquartile range 25–35) and an average of 31.34 (±9.37), which represents “poor quality.” 48% of the videos received “very poor,” 38% received “poor,” 10% received “fair” and 4% received “good” ratings, with none achieving an “excellent” rating. Overall, the highest rated question was “was it relevant?” and the lowest was “does it provide support for shared decision-making?” [Table 1]. The linear regression showed a positive correlation between the run time and DISCERN score (r = 0.71, P < 0.001) and a negligible correlation between the number of views and mean DISCERN score (r = −0.03799, P = 0.79).

Table 1

Mean DISCERN scores with standard deviations of all 50 videos based on the 3 evaluators in this study

Discern question	Average ratings
Is it relevant?	3.21±0.63
Is it balanced and unbiased?	2.89±0.68
Does it describe how each treatment works?	2.52±0.91
Does it achieve is aims?	2.42±1.27
Are the aims clear?	2.32±0.89
Is it clear that there may be more than one possible treatment choice?	2.22±1.17
Does it describe the benefits of each treatment?	2.17±0.90
Does it describe the risks of each treatment?	1.82±0.95
Does it describe how the treatment choices affect overall quality of life?	1.63±0.67
Does it describe what would happen if no treatment is used?	1.56±0.74
Does it refer to areas of uncertainty?	1.56±0.73
Is it clear what sources of information were used to compile the publication?	1.23±0.70
Does it provide details of additional sources of support and information?	1.20±0.45
Is it clear when the information used or reported in the publication was produced?	1.18±0.63
Does it provide support for shared decision-making?	1.13±0.35

Patient education materials assessment tool for audiovisual materials

To examine the understandability and actionability of the videos, PEMAT-AV was used. The overall average understandability of the videos was 69.8% ±15.4%, with 70% representing the cut-off for achieving understandability. A little over half of videos (56%) achieved this standard. Table 2 shows the results of the PEMAT-AV understandability ratings. Most of the videos did not have a summary nor had informative headers as most of these videos had little to no postproduction. In addition, some of the videos had difficulty using only common, everyday language, or not explaining the medical terms used in the videos.

Table 2

Mean Patient Education Materials Assessment Tool for Audiovisual Material understandability scores with standard deviations of all 50 videos based on the 3 evaluators in this study

PEMAT-AV understandability question	Average ratings (%)
Text on the screen is easy to read	98.6±12.2
The material presents information in a logical sequence	98.0±14.2
The material uses the active voice	98.0±16.0
The material allows the user to hear the words clearly (e.g., not too fast, not garbled)	97.7±10.7
The material makes its purpose completely evident	75.0±34.8
The material breaks or “chunks” information into short sections?	66.3±39.3
The material uses common, everyday language	66.0±41.1
Medical terms are used only to familiarize audience with the terms. When used, medical terms are defined	63.0±41.8
The material’s sections have informative headers	33.7±43.5
The material provides a summary	12.2±29.3

PEMAT-AV=Patient Education Materials Assessment Tool for Audiovisual Material

Table 3 displays the results of the PEMAT-AV actionability. PEMAT-AV actionability ratings scored much poorer, with an average of 11.0% ±24.6%, with 60% as the cut-off for actionability. This cut-off was adapted because the last section of the actionability tool was not applicable (“The material explains how to use the charts, graphs, tables, or diagrams to take actions.”) Only 5 (10%) of the videos achieved this score. Actions such as subscribing to the channel were not included as pertinent actionability items as they have no clinical significance.

Table 3

Mean Patient Education Materials Assessment Tool for Audiovisual Material actionability scores with standard deviations of all 50 videos based on the 3 evaluators in this study

PEMAT-AV actionability question	Average ratings (%)
The material clear identifies at least one action the user can take	31.3±32.9
The material breaks down any action into manageable, explicit steps	10.7±21.5
The material addresses the user directly when describing actions	9.3±19.7

PEMAT-AV=Patient Education Materials Assessment Tool for Audiovisual Material

To assess the robustness of our study, the IRR score was calculated. Table 4 represents all of the IRR scores based on the two tools used with the PEMAT-AV understandability and actionability scores separated.

Table 4

The interrater reliability scores for the DISCERN, Patient Education Materials Assessment Tool for Audiovisual Material understandability, and Patient Education Materials Assessment Tool for Audiovisual Material actionability scores between the 3 evaluators for all 50 videos in this study

Raters	DISCERN	PEMAT-AV understandability	PEMAT-AV actionability
JL - SD	0.545	0.775	0.650
JL - SH	0.715	0.925	0.925
SD - SH	0.513	0.780	0.680

PEMAT-AV=Patient Education Materials Assessment Tool for Audiovisual Material, JL=Jacob Lang, SD=Shivashankar Damodaran, SH=Stephen Hong

Because of the large percentage of videos made outside of the United States (36%), a comparison between the origins of these producers is represented in Figure 1. The majority of these international producers were from India (83%). A comparison of the two sources of information shows that the international videos had a mean run time of 7 min and 2 s, while the US videos ran for 5 min and 49 s. Although the analysis showed that the international producers had higher averages in all three categories, none of the P values from the homoscedastic t-test were statistically significant. These P values were 0.28, 0.67, and 0.20 for the DISCERN, PEMAT-AV understandability, and PEMAT-AV actionability scores, respectively.

Figure 1

A comparison of the mean PEMAT-AV and DISCERN scores with standard deviations of video producers from the United States versus the rest of the World, showing a statistically insignificant higher score in the international PEMAT-AV and DISCERN scores

DISCUSSION

There has been a growing movement in the urological community to address the increase in use of social and digital media to engage the patient population.[2223] Although patients still highly trust the information from their physicians, there is an increasing reliance on the internet to gather health-related information.[24] Therefore, it is imperative that an objective analysis of medical information, including that in the urological field, be conducted on a periodic basis. This is the first study to examine the quality of the information on varicoceles provided on YouTube for the consumption by the general public.

Utilizing the first 50 nonduplicated videos, this study encompasses more than 5 h of videos with over 5 million views. The overwhelming majority of these videos were meant to be educational with only 4% of the videos representing “personal experiences” or “blogs.” Based on this study, the quality of the information provided in these videos were rated as “poor” on the DISCERN tool. Understandably, longer video scored higher in the DISCERN tool because of the time needed to provide reliable information. However, there was no correlation between the number of views and the quality of the video. This is of concern as it would be ideal that higher-quality videos would be accessed more often than those of lower quality.

Many of these videos seemed to lead patients into actively seeking treatment through the various methods available for varicocele, often focusing on research indicating varicocele as the most common known causes of male infertility within the videos.[142526] With a large portion (60%) of the content creators being physicians from a private practice, they may have a more vested interest in convincing their patients that surgical treatment is the only and best method improving male fertility. In addition, they often fail to provide opportunities for the viewer to search additional resources or provide the support for shared-decision making with their health provider. Luckily, all but one content creator provided accurate information. It will be an ethical challenge to combat misleading health information as the goal of providing ease of access to information is meant to be inclusive to the entire population. Alienation to certain groups because of health misinformation can be extremely polarizing, as seen with “anti-vaxxers.[27]”

Although there are major pitfalls in the informational quality of the videos, a strength was found in that they generally rated well in the understandability of information provided. The high PEMAT-AV understandability scores of these videos can be attributed to the visual appeal, the abundance of cheap, high-quality audio and video equipment, and the ease of use for postproduction programs on the market. Many of these videos could have been recorded using a mobile telephone device, edited using free programs like iMovie on Apple products, and still be able to achieve high scores on DISCERN and PEMAT-AV.

With YouTube's international market penetration, it seems that the same pitfalls that the American medical video producers suffer from are shared with their international counterparts, represented by the statistically insignificant difference between the mean DISCERN and PEMAT-AV scores. However, the quality of these videos tended to be slightly higher, which can be correlated to the longer run times in the international videos. The overwhelming majority of the international videos were from India. With China having a ban on all Alphabet products, the parent company of YouTube, India is the largest country by population to upload and view content on YouTube's platform. There are an estimated 342.3 million viewers in India, which surpasses the current US population, making it unsurprising that Indian producers made up the majority of the international population.[28]

This study is not without limitations, most pertinently the massive amount of information being constantly uploaded on the YouTube platform and some of the subjectivity of the DISCERN tool. This study must be defined on a specific timeline, while the amount of information and the YouTube algorithm to rank videos on its website remains fluid. This fluidity will require periodic checks to assess the evolution of information quality. An additional limitation is the DISCERN tool's flexibility to allow some rater interpretation of a video's ability to satisfy each quality criterion by giving partial scores between 2 and 4. Regardless, these scores and the PEMAT-AV scores were shown to have good robustness as the IRR scores ranged from 0.513 to 0.925, indicating a “moderate” to “almost perfect” kappa scores.[29] This and future studies that continue to examine the quality of new information being uploaded on the platform can provide guidance for content creators to address the drawbacks of the current information and take in considerations of providing a more holistic view of the topic of varicoceles.

As YouTube's popularity increases for health education and information, it is important that individual medical specialties and the medical community as a whole work to assess the quality of this information. This is not usually done by YouTube's parent company Google, as their videos are listed usually on popularity and other proprietary algorithms that do not take scientific analysis into account. It is in the opinion of the authors that if the most subscribed individual YouTube content creator, PewDiePie, were to create a video on varicoceles, it would be the first video option in searches related to this topic due to his over 108 million subscribers, regardless of the validity of the content in this hypothesized video. This creates challenges in the patient population to accessing quality and verified information about health problems. However, this provides an opportunity for the medical and scientific community to embrace this method of information dissemination and as an opening to work with these large broadcasting platforms.

CONCLUSIONS

We found that the quality of the information on varicoceles on YouTube is poor. There is a need to develop videos that approache varicoceles more holistically that not only inform patients, but also empower them to make the decision that best reflects their goals in treatment.