Quality and reliability evaluation of online videos on carpal tunnel syndrome: a YouTube video-based study.

OBJECTIVES: With the increasing popularity of searches for medical information on YouTube, the availability of videos concerning carpal tunnel syndrome (CTS) is increasing. This study aimed to evaluate the quality and reliability of YouTube videos on CTS. SETTING AND PARTICIPANTS: No participants were included. PRIMARY AND SECONDARY OUTCOME MEASURES: We searched YouTube on 1 April 2021 using the keywords "carpal tunnel syndrome" and "carpal tunnel release" and evaluated the first 55 retrieved videos. We summarised the video characteristics including Video Power Index (VPI), which was designed to evaluate video popularity based on the number of likes and views. We categorised them based on source and content. Video quality and reliability were evaluated using the Journal of the American Medical Association (JAMA) benchmark criteria, the Global Quality Score (GQS) and the Carpal Tunnel Syndrome-Specific Score (CTS-ss) .
RESULTS: The mean (range: minimum-maximum) of JAMA scores, GQS and CTS-ss were 2.13 (1-4), 2.69 (1-5), and 5.0 (1-15), respectively. The most common source of video was from allied health workers, and academically sourced videos had the highest JAMA score and GQS. Three scores were significantly correlated with each other. Multiple linear regression analysis showed that a higher JAMA score was associated with a higher likes ratio, and a higher GQS was associated with a longer video running time and greater number of comments. However, a higher VPI was not associated with higher video quality or reliability represented by the three scores.
CONCLUSIONS: YouTube videos on CTS have low quality and reliability. Video popularity was not significantly correlated with quality or reliability. Our findings suggest that expert groups should provide and promote high-quality video content to YouTube users and patients. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Various characteristics including number of views, number of likes, Video Power Index and video uploader in the YouTube videos about carpal tunnel syndrome were investigated.

The reliability and quality of videos were investigated using three scoring systems: Journal of the American Medical Association benchmark criteria, Global Quality Score and Carpal Tunnel Syndrome-Specific Score.

Although these scoring systems are subjective and unvalidated, the scoring systems were independently assessed twice by the two raters, which showed intraobserver and interobserver agreements determined by intraclass correlation coefficients.

A multiple linear regression analysis was performed to identify video characteristics affecting the reliability and quality of videos.

Background

With the internet penetration rate exceeding 50% worldwide,1 searches for health information on the internet have become common. According to recent studies, 80% of internet users searched for health information online,2 and up to 30% of orthopaedic patients searched online for disease information.3 Furthermore, well-designed videos of disease information positively affect treatment outcomes by improving patient comprehension.4 5 However, most online information is not regulated, resulting in the spread of inaccurate and low-quality data among patients.6–10 Therefore, physicians should properly evaluate such information and help patients obtain accurate information and appropriate treatment.

YouTube, which has over 1 billion users watching over 1 billion hours of videos each day, is a source of representative video-based educational content.11 Although some high-quality orthopaedic content is uploaded by qualified experts on YouTube, most of the related content is uploaded by unqualified individuals, providing patients inaccurate and erroneous information. In previous quality-evaluating studies in the orthopaedic field, YouTube video accuracy and quality were low.1 10 12–14

According to previous studies that investigated the quality of carpal tunnel syndrome (CTS) information provided by internet search engines,15–17 the quality of online information has improved over the past decade but remains low. These studies reported that there was significant scope for improvement. In contrast, recent studies18 19 reported that most YouTube videos and websites that provide information on CTS can reinforce misconceptions. Two quality-evaluating studies on CTS information available on YouTube20 21 focused on video quality and reliability, and neither examined the relationship between characteristics such as video popularity and quality.

The current study aimed to (1) evaluate the quality and reliability of YouTube videos concerning CTS; (2) investigate the video characteristics, sources and contents; and (3) determine the relationship between video characteristics and quality.

Methods

Patient and public involvement

No patients were involved.

YouTube search design and study setting

The YouTube online library (https://www.youtube.com) was searched on 1 April 2021 using the terms “carpal tunnel syndrome” and “carpal tunnel release”. The first 50 videos retrieved based on each keyword and sorted by ‘view count’ for a total of 100 videos were selected for review. Of them, 45 were excluded (duplicates, 39; non-English, 3; information on cubital tunnel syndrome, 2; and soundtrack with no mention of carpal tunnel, 1). Thus, 55 YouTube videos found using the keywords “carpal tunnel syndrome” and “carpal tunnel release” were analysed (figure 1). The URLs of each video are listed in online supplemental table 1.

Figure 1

Search methodology for carpal tunnel syndrome-related YouTube videos.

Data on the following video characteristics were collected from each YouTube video: (1) title, (2) channel name, (3) number of subscribers, (4) video running time, (5) number of views, (6) number of comments, (7) video source/uploader, (8) content type, (9) days since upload, (10) view ratio (number of views:days since upload), (11) number of likes, (12) number of dislikes, (13) likes ratio (likes×100/[likes+dislikes]) and (14) Video Power Index (VPI). The VPI was calculated using the following formula: like ratio×view ratio/100. This value is an index designed to evaluate video popularity based on the number of likes and views.1

Video sources/uploaders were categorised as follows1 10: (1) academic (uploaders affiliated with universities or research groups); (2) physicians (individual physicians or physician groups not affiliated to a university or research institute); (3) non-physicians (allied health workers such as alternative medical providers, physiotherapists and occupational therapists); (4) trainers; (5) medical sources (animations or related content from health websites); (6) patients; and (7) commercial. Contents were categorised as follows: (1) exercise training; (2) disease information; (3) patient experience; (4) surgical technique; (5) non-surgical management, such as chiropractic treatment; and (6) advertisement.

Evaluation of video quality and reliability

The quality and reliability of YouTube videos were assessed using three scoring systems: the Journal of the American Medical Association (JAMA) benchmark criteria, the Global Quality Score (GQS) and the Carpal Tunnel Syndrome-Specific Score (CTS-ss). The JAMA criteria enable a non-specific assessment of content reliability and include four criteria (table 1).22 Each criterion is assigned 1 point for a maximum total of 4 points. A score of 0 indicates low video reliability and accuracy, whereas a score of 4 indicates high video reliability and accuracy. The GQS1 10 23 consists of five grades and provides a non-specific assessment of health-related website quality (table 2). The total GQS ranges from 1 to 5, with a higher score indicating better educational quality. To better evaluate quality and accuracy of YouTube videos concerning CTS, we employed the new CTS-ss, which consists of 20 items. We generated this scoring system based on recent review articles24–26 and guidelines published by the American Academy of Orthopaedic Surgeons,27 which were considered reasonable in previous studies.9 10 The CTS-ss evaluates information on (1) patient symptoms and population, (2) carpal tunnel anatomy, (3) CTS diagnosis and evaluation, (4) treatment options, and (5) postoperative care and course (box 1). One point was given for each of the 20 items for a total maximum of 20 points. Higher scores indicated higher CTS-specific educational value.

Table 1

Journal of the American Medical Association benchmark criteria22

Criterion	Description
Authorship	Author and contributor credentials and their affiliations should be provided.
Attribution	All copyright information should be clearly listed, and references and sources for content should be stated.
Currency	The initial date of posted content and dates of subsequent updates to content should be provided.
Disclosure	Conflicts of interest, funding, sponsorship, advertising, support and video ownership should be fully disclosed.

Table 2

Global Quality Score criteria1 10 23

Grade	Description of quality
1	Poor quality, information missing, technique misleading; unlikely to be useful for patient education
2	Generally sparse quality, some information provided but majority lacking, technique poor; limited use for patients
3	Moderate quality, important information provided but some lacking, technique mostly adequate; somewhat useful for patients
4	Good quality, majority of information provided but some information lacking, technique adequate; useful for patients because most important topics are covered
5	Excellent quality, full information provided, technique adequate; highly useful for patients

Describes symptoms (eg, nocturnal paraesthesia, loss of sensation, thenar muscle atrophy).

Describes patient population, especially high prevalence in older women.

Describes carpal tunnel anatomy and/or function.

Mentions caused by nerve compression.

Describes risk factors (eg, diabetes, hypothyroidism, pregnancy and repetitive use).

Mentions physical examination and findings (eg, Tinel’s sign and Phalen’s manoeuvre).

Discusses electrophysiological tests.

Discusses additional diagnostic tests (eg, ultrasound and MRI).

Mentions patient-centred measures (eg, the Boston Carpal Tunnel Syndrome Questionnaire).

Discusses differential diagnosis (eg, cervical radiculopathy).

Describes non-surgical treatment, especially changes in habits.

Mentions that laser therapy is one of the non-surgical options.

Mentions pharmacotherapy (eg, local corticosteroid injection, NSAIDs).

Mentions musculoskeletal manipulation and/or splinting.

Describes surgical treatment that is the most effective treatment.

Mentions open carpal tunnel release.

Mentions endoscopic carpal tunnel release.

Describes complications and outcomes (eg, CRPS, scar tenderness, reoperation).

Mentions need for postoperative physical therapy.

Outlines return-to-function timeline.

Intraobserver reliability and interobserver agreement assessment

All three scoring systems (JAMA, GQS and CTS-ss) were independently assessed twice, 30 days apart, by two raters, consisting of one orthopaedic surgeon (DK) and one family medicine doctor (YK). Intraobserver and interobserver agreements were determined using intraclass correlation coefficients (ICCs). ICCs for absolute agreement with a single measurement were used to identify intraobserver reliability with a two-way mixed-effects analysis of variance models. ICCs for absolute agreement with a single rater were used to identify interobserver agreement using two-way random-effects analysis of variance models. A guideline28 for evaluating ICC values was adopted: excellent (>0.90), good (0.75–0.90), moderate (0.50–0.75) and poor (<0.50). In cases of disagreement, all authors re-evaluated the video in question until consensus was reached.

Statistical analysis

Continuous variables are presented as mean±SD. Differences in the JAMA score, GQS, CTS-ss and VPI according to (1) video upload source and (2) category of video contents were evaluated by one-way analysis of variance tests (for normally distributed data) and Kruskal-Wallis tests (for non-normally distributed data) followed by post hoc tests using the Bonferroni method. A Spearman correlation analysis was used to assess the correlation between scores and between video characteristics and scores. A multiple linear regression analysis was performed to identify video characteristics affecting the JAMA score, GQS, CTS-ss and VPI. All reported p values were two-sided, and those <0.05 were considered statistically significant.

Results

Video characteristics and quality scores

The mean JAMA score, GQS and CTS-ss were 2.13, 2.69 and 5.0, respectively, indicating low reliability and educational quality (table 3). Raw scores of JAMA score and CTS-ss are shown in online supplemental table 2. Non-physician video sources accounted for the largest share (29.09%), while commercial sources accounted for the lowest share (5.45%) (figure 2). Disease-specific information accounted for the largest share (32.73%), while patient experience accounted for the smallest share (3.64%) (figure 3). The video title, YouTube channel name, JAMA score, GQS, CTS-ss and VPI of the top 55 videos are listed in order of the number of views in figure 4.

Table 3

Characteristics of 55 YouTube videos about carpal tunnel syndrome

Variable	Value
Number of subscribers	742 791.7±1 183 968
Video running time (s)	400.71±271.91
Number of views	1 559 722±7 629 661
Number of days since upload	2450.27±1250.96
Number of comments	316.75±332.4
Number of likes	5184.51±4804.72
Number of dislikes	242.8±421.93
View ratio	478.77±1506.85
Like ratio	92.81±7.39
VPI	382.9±910.34
JAMA scores	2.13±0.94
GQS	2.69±1.17
CTS-ss	5.0±3.29

Data are presented as mean±SD.

Formulas: view ratio, number of views/days since upload; like ratio, number of likes×100/[number of likes+number of dislikes]; VPI, like ratio×view ratio/100.

CTS-ss, Carpal Tunnel Syndrome-Specific Score; GQS, Global Quality Score; JAMA, Journal of the American Medical Association; VPI, Video Power Index.

Figure 2

Categorical distribution of video source.

Figure 3

Categorical distribution of video content.

Figure 4

Data-bar visualisation of the top 55 carpal tunnel syndrome and release videos with the highest number of views. CTS-ss, Carpal Tunnel Syndrome-Specific Score; GQS, Global Quality Score; JAMA, Journal of the American Medical Association; VPI, Video Power Index.

Differences in video reliability and quality by source and content

The JAMA score (p<0.0001) and GQS (p=0.0004) differed significantly among the seven groups of video sources, with videos from academic and physician sources having the highest mean JAMA scores and GQS (table 4). The JAMA score (p=0.0077) and GQS (p=0.0018) differed significantly among the six groups of video content, with videos about surgical technique and disease-specific information having the highest mean JAMA scores and GQS. However, the CTS-ss and VPI did not differ significantly between the groups based on video sources and contents.

Table 4

Mean quality and reliability scores per video source and video content variable

Grouping variable	JAMA score	GQS	CTS-ss	VPI
Video source
Academic	3.38±0.74	3.63±1.06	6.12±5.0	1077.92±2324.16
Physician	2.7±0.82	3.5±1.18	6.4±3.24	156.50±79.12
Non-physician	2.0±0.52	2.43±0.73	4.13±2.28	314.65±204.90
Trainer	1.25±0.5	1.5±0.58	3.0±2.31	243.20±157.61
Medical	1.7±0.82	2.7±1.25	5.6±3.41	371.63±370.09
Patient	1.25±0.5	1.25±0.5	2.25±0.5	172.21±127.05
Commercial	1.33±0.58	2.33±0.58	6.33±3.06	152.93±122.48
P value*	<0.0001	0.0004	0.1306	0.4234
Significant difference in post hoc analysis†	Academic versus non-physician, trainer, medical, patient, commercial;Physician versus trainer, medical, patient, commercial	Academic versus trainer, patient;Physician versus trainer, patient
Video content
Exercise training	1.73±0.79	1.91±0.83	3.09±1.97	344.15±266.65
Disease-specific	2.33±0.84	3.17±1.04	6.22±3.54	227.41±161.24
Patient experience	1.5±0.71	1.5±0.71	2.5±0.71	133.82±109.52
Surgical technique	2.83±1.11	3.42±1.16	5.92±3.65	724.92±1917.21
Non-surgical	1.63±0.52	2.13±1.13	4.13±2.64	396.44±367.10
Advertisement	1.5±0.58	2.25±0.5	5.0±3.65	260.57±237.37
P value‡	0.0077	0.0018	0.0897	0.3493
Significant difference in post hoc analysis†	Surgical technique versus exercise training, non-surgical	Disease-specific, surgical technique versus exercise training

Data are presented as mean±SD.

*For the video source group, significant differences were seen in JAMA score and GQS.

†Post hoc tests were performed using Bonferroni’s method.

‡For the video content group, significant differences were seen in JAMA score and GQS.

CTS-ss, Carpal Tunnel Syndrome-Specific Score; GQS, Global Quality Score; JAMA, Journal of the American Medical Association; VPI, Video Power Index.

Factors affecting video quality and popularity

JAMA, GQS and CTS-ss significantly correlated with each other (JAMA score vs GQS, p<0.001; JAMA score vs CTS-ss, p=0.001; GQS vs CTS-ss, p<0.001). However, the VPI was not significantly correlated with the three scores. Multiple linear regression analysis showed that a higher JAMA score was associated with a higher likes ratio of an academic or physician upload source compared with a patient upload source (table 5). A higher GQS was associated with a longer video running time; greater number of comments; and higher probability of academic, physician, non-physician, medical information and commercial upload source than of patient upload source. A higher CTS-ss was more associated with academic, physician, medical information, and commercial upload sources than patient upload sources. However, a higher VPI was not associated with higher video quality or reliability scores.

Table 5

Multiple linear regression analysis of correlations between video characteristics and the VPI, JAMA score, GQS and CTS-ss

Variable	Unstandardised beta (B)	95% CI	Standardised β	P value
VPI (R2=0.997)
Days since upload	−0.039	−0.058 to −0.02	−0.053	<0.001
View ratio	0.595	0.576 to 0.614	0.985	<0.001
Number of likes	14.118	6.808 to 21.428	0.075	<0.001
JAMA score (R2=0.626)
Like ratio	0.054	0.001 to 0.107	0.424	0.045
Video source
Academic	2.126	1.164 to 3.088	0.801	<0.001
Physician	1.187	0.239 to 2.136	0.49	0.015
GQS (R2=0.561)
Video running time	0.001	0 to 0.002	0.252	0.044
Number of comments	0.002	0 to 0.003	0.461	0.029
Video source
Academic	3.025	1.735 to 4.315	0.921	<0.001
Physician	2.465	1.193 to 3.736	0.821	<0.001
Non-physician	1.596	0.337 to 2.856	0.626	0.014
Medical	1.878	0.661 to 3.094	0.625	0.003
Commercial	1.874	0.32 to 3.429	0.368	0.019
CTS-ss (R2=0.356)
Video source
Academic	6.225	1.825 to 10.624	0.673	0.007
Physician	5.174	0.838 to 9.51	0.612	0.021
Medical	4.978	0.828 to 9.128	0.589	0.02
Commercial	6.430	1.13 to 11.731	0.448	0.019

CTS-ss, Carpal Tunnel Syndrome-Specific Score; GQS, Global Quality Score; JAMA, Journal of the American Medical Association; VPI, Video Power Index.

The intraobserver reliability of the two raters was excellent for the JAMA score, GQS and CTS-ss. The interobserver agreement between raters was good for the JAMA score (ICC 0.881, 95% CI 0.804 to 0.929), good for the GQS (ICC 0.881, 95% CI 0.804 to 0.929) and excellent for the CTS-ss (ICC 0.941, 95% CI 0.898 to 0.966).

Discussion

This study demonstrated that the reliability and quality of YouTube videos concerning CTS were low. This result was consistent with that of other previously conducted YouTube video quality evaluation studies.1 10 13 20 21 29–31 Mert and Bozgeyik20 evaluated the quality of CTS videos on YouTube and reported that the video reliability and quality were low. They presented no significant relationship between video characteristics, reliability and quality evaluation scoring systems. Radonjic et al21 also evaluated CTS videos on YouTube and showed low reliability and quality and found that videos uploaded by physicians had significantly higher reliability and quality evaluation scores than those uploaded by non-physicians. Goyal et al18 reported that YouTube videos of CTS have low information quality. They determined that the potential reinforcement of misconceptions is prevalent in YouTube videos on CTS.

Although the overall reliability and educational quality of YouTube videos were low, those of videos from academic and physician uploaders or about surgical techniques and disease-specific information were significantly higher than those of other video sources and contents. This is because the main purpose of these video sources and contents is to educate doctors, medical students and patients. In contrast, the CTS-ss did not differ significantly among the video sources and contents because YouTube videos focus on specific topics, such as symptoms and surgical technique or rehabilitation after surgical treatment, and deliver the content within a short running time. Additionally, some specific channels, such as the ‘Bob & Brad’ channel, posted videos in four series about CTS and release. Casual YouTube viewers cannot obtain sufficient content on CTS and release in only one or two posted videos, but an entire series can provide most of the content. YouTube uploaders usually post short videos of less than 10 min to maximise the number of views and user interest; thus, they split the content into several videos.

Most of the videos had low reliability and educational quality, but some videos had useful practicality and educational information. The ‘Carpal Tunnel Syndrome - Everything You Need To Know - Dr. Nabil Ebraheim’ video of the nabil ebraheim channel explains the overall symptoms, anatomy and risk factors of CTS. In the ‘Surgery Video: Carpal Tunnel - MedStar Union Memorial’ video of the MedStar Health channel, the surgical procedure and method of endoscopic carpal tunnel release were shown in detail. The ‘How to Determine If You Really Have Carpal Tunnel Syndrome - Dr Mandell, DC’ video of the motivationaldoc channel shows the physical examination required for CTS diagnosis.

In this study, video popularity showed no significant correlation with reliability or quality. Popular videos that casual YouTube users and patients frequently watch do not have good quality and reliability. Interestingly, YouTube videos of expert groups that are expected to have high reliability and quality, such as the American Academy of Orthopaedic Surgeons or Federation of European Societies for Surgery of the Hand, were not included in the top 55 videos. A manual search identified only about 1600 views for the carpal tunnel release video uploaded to the American Academy of Orthopaedic Surgeons YouTube channel (https://www.youtube.com/watch?v=eemuH5UYElo). Additionally, the Federation of European Societies for Surgery of the Hand and the British Society for Surgery of the Hand channels have no CTS-related videos and only 154 and 575 subscribers, respectively. It is necessary to promote an expert group’s YouTube videos and channels and try to provide accurate medical information by uploading a high-quality video and exposing it to casual YouTube users and patients.

In a previous study on the meniscus,10 video dislikes were described as predictors of YouTube video reliability, but this was not the case in this study. The independent predictor of the JAMA score in this study was the likes ratio. Furthermore, independent predictors of GQS were video running time and number of comments, suggesting that videos with a longer running time and greater number of comments are independently and significantly associated with a higher GQS. The longer the video running time, the greater the amount of information it contains; therefore, its educational quality also increases. For GQS, a greater number of comments contain more useful information for users who watched the video. Regarding the CTS-ss, compared with patient upload sources, academic, physician, medical and commercial upload sources are associated with a higher CTS-ss. However, unlike the JAMA score and GQS, CTS-ss showed no significant association with video characteristics except for video source.

Our study has several limitations. First, we searched the top 50 videos for “carpal tunnel syndrome” and “carpal tunnel release” on YouTube in the order of popularity. This search strategy missed certain videos with low views or hits but with potentially high quality. Although our search strategy could miss high-quality videos that are less ‘popular’, this strategy is the actual method by which casual YouTube users obtain information. Second, YouTube video metrics such as the number of likes and views are constantly updated; therefore, these study data are accurate only on the date of the search. Third, the assessment scoring systems that we used (the JAMA score, GQS and CTS-ss) are subjective and unvalidated. Because the JAMA benchmark criteria were developed to assess medical information on the internet website rather than video information, the criteria may not fit YouTube videos. The CTS-ss includes many contents of CTS, but almost YouTube videos have a short duration of about 10 min or less. Thus, it tends to be difficult to present all checklist of CTS-ss in short videos. Because some criteria in the JAMA benchmark criteria and CTS-ss were unsatisfied in most of videos, total score may be mainly influenced by some criteria; thus, all criteria have no equal weight. Nevertheless, we have no choice but to use these scoring systems due to lack of validated scoring system for evaluating the quality and reliability of medical information in YouTube videos. The excellent interobserver and intraobserver reliability were confirmed using ICCs to redeem these shortcomings. In addition, the GQS may be highly subjective; thus, we tried to resolve the subjectivity by having two independent authors perform each evaluation twice. Fourth, one video entitled ‘Podcast: See a Live Surgery for Carpal Tunnel Syndrome’ has the dominant number of views (66.5%), so the average views and VPI values tended to increase. We tried to buffer this dominance by analysing 55 videos.

Conclusions

This study demonstrated that YouTube videos of CTS showed low reliability and quality. Video quality is significantly associated with content and upload source. Video popularity was not correlated with video reliability or quality, which suggests that a good content quality does not guarantee video popularity. The impact of videos on patient care cannot be underestimated. To ensure the spread of accurate information, it is necessary to have YouTube videos published by expert groups and to strive to provide high-quality video materials that can assist with patient diagnosis and treatment.