Measuring the impact and reach of informal educational videos on YouTube: The case of Scientific Animations Without Borders.

Given the popularity, reach, and variable accessibility of online platforms as channels for informal education by higher-education institutions (HEIs), it becomes practically and theoretically important to better understand the factors that affect the impact and reach of any such Internet-delivered ICT informal learning. Accordingly, this study analysed viewer data from one informal, science animation educational channel on currently the most-watched online platform, YouTube, to measure characteristics affecting the videos' impact and reach. Results from the study identified the most watched videos on the channel-including survival gardening using drip irrigation, charcoal water filtration, and tuberculosis prevention-and characteristic demand, time, location, and volume of video access. While the basic findings reaffirm prior research measuring statistically significant correlations between user-activity and YouTube engagement metrics (including watch-time, view counts, likes, and subscribers), they also demonstrate the and ability to make such content broadly applicable across demographics and globally. Offering some measure of insights for boosting HEI participation in informal educational science animation and ICT learning practices using platforms like YouTube-especially around the topics of water, agriculture, food security, and improved health outcomes-the findings also point to where improved methods for reaching the intended recipients of informal education online are needed to best leverage the potential of such platforms.

Introduction

Background

While Internet-based information and communication technology (ICT) approaches have radically altered the potential for education and the delivery of information to people in every corner of the world, basic questions remain whether information delivered is being received and acted upon by recipients. Given that one UNESCO document notes, “Understanding ICT and mastering the basic skills and concepts of ICT are now regarded by many countries as part of the core of education alongside reading and writing” (Anderson et al., 2002, p. 1), achieving successful delivery, reception, and buy-in of digital educational material by would-be recipients is becoming more and more essential.

Unfortunately, barriers to ICT education (particularly within less digitally saturated countries) can significantly limit the reach of this needed access. These limitations include but are not limited to (1) developmental shortfalls (around technology, resources, and infrastructure), (2) access barriers (particularly in remote or rural regions of countries), (3) resistance to ICT practices or a failure to integrate them into education by educators and school administrations, (4) linguistic barriers and socio-historical hierarchies of dialects (particularly in highly multilingual regions, like Africa), and (5) socioeconomic and cultural barriers that preclude access by certain people (often women and girls) even after developmental barriers have been overcome (Bello-Bravo et al., 2019a, Bello-Bravo et al., 2019b; Kiramba, 2018; Sharma, 2003; Tsai and Chai, 2012).

As a result, recurrent challenges that face Internet-based ICT education include (1) uncertainty by providers whether an educational message is even reaching its intended audience, (2) whether, if it is reaching its audience, it is being retained and acted upon by the recipients hearing it, and (3) who is being missed or not receiving the message at all, despite its otherwise successful transmission into an area. Of these three challenges, the second is ubiquitous across all educational settings and implicates what it means to “teach” in the broadest sense (Conley, 2014); this study does not address this perennial challenge. The first and third challenges, however, are both measurable and characterizable.

For the first challenge, the issues are largely technocratic and technological, with the caveat that successful implementation involves more than simply the necessary material means for information transmission (i.e., more than only adequate electrical and Internet infrastructures, ICT-sending and receiving devices, and human support structures to keep these affordances operating stably, much of which is often missing or in very short supply in remote areas and digitally less saturated national contexts (Bello-Bravo and Pittendrigh, 2018). As such, “cultural” technologies are needed as well—above all, translating any educational material into the intended audience's locally most comfortably spoken dialect (Bello-Bravo and Pittendrigh, 2018; Nonaka and Takeuchi, 1997; Szulanski et al., 2004). For example, a YouTube educational video transmitted into a person's home or to their cellphone represents an only partial success for delivery when the dialect of that video is not also one that the recipient understands (Lidofsky et al., 2019; Park, Ton, Yeo et al., 2019). Moreover, any translation is rarely linguistic-only but also requires “translating” cultural behaviors and modes of being as well (Ajiboye, 2016; Kelly, 1979; Metleaeva, 2016), which often is the most difficult part (Braçaj, 2015). As such, while overdubbing the video into the intended audience's most comfortably spoken dialect is an absolutely basic element for enhancing the likely success of an educational video, further cultural factors can still affect reception. For example, message media (especially visual content) that seem to parody or misrepresent the target recipients' identities can be off-putting and lead to a diminishment of the message's value, if not its rejection outright (Bello-Bravo, 2011; Bello-Bravo et al., 2011; Hoffman, 2011); for this reason, live-action compared to animated educational content can risk detrimental impacts on learning (Moreno and Ortegano-Layne, 2008; Smith et al., 2012). Taken together, these material and cultural factors represent “affordances” essential to successful ICT message delivery.

These material and cultural affordances also play a role for the third challenge as well. Here, the problem is less that the message failed to be delivered and more that some people (often women and girls) are missed or simply excluded as potential message recipients. This well-documented gender gap in ICT delivery and use (Gillard et al., 2008; Hafkin, 2000; Manfre, 2011) is echoed in a rural/urban access gap as well (Bunyi, 2008; Hindman, 2000). As such, typically resource-straitened national education efforts may lack the means to substantially or effectively extend resources beyond majority populations in urban centers. Whether this prioritization of the urban over the rural is a willful expression of hierarchical social power (Bunyi, 2008; Kiramba, 2018) or an unfortunate consequence born from a shortage of resources, the result in either case is that critical public education issues are not communicated to everyone who needs to (or would benefit from) hearing them.

A public messaging campaign that reaches fewer of its intended audience members by definition performs more poorly than one that reaches more of its intended audience (Ajzen and Fishbein, 1980; Atkin, 2001). For this reason alone, identifying effective strategies for overcoming technological, economic, and cultural hindrances to message delivery becomes necessary for achieving the best possible outcomes from public messaging. One of the most promising of these strategies includes delivering locally translated, educational/scientific videos to recipients’ video-enabled cellphones (through sharing via Bluetooth, access to YouTube or other online platforms, and other digital means) (Bello-Bravo et al., 2019). This potential of cell phones—as now the most prevalent and technologically familiar means of Internet access (Bello-Bravo et al., 2021), even in remote locations within otherwise digitally less saturated contexts)—is immense, but still has important limitations across male/female and rural/urban usage gaps (Bello-Bravo et al., 2017; Eubanks, 2012; Hafkin, 2000). And although these recurrent challenges both require solutions and are already to some extent addressed by the affordances of cell phones themselves for delivering educational messages more broadly than other ICTs (Bello-Bravo and Pittendrigh, 2018), a significant bottleneck occurs when messages must be accessed (in the first place) via the Internet.

Where Internet infrastructures are available—including availability in remote locations within digitally less saturated regions—direct access to informal educational videos becomes possible through YouTube (and other web-based platforms). As the second-most visited website worldwide (BrandsDistribution, 2019), YouTube continues to grow as one of the most relevant mass communication media platforms. YouTube channels have enriched access to information and informal education through scientific videos by allowing (Internet) users free access to scientific content. With an estimated 1 billion hours of video watched daily by some 63 million viewers worldwide, YouTube is the leading access point on the Internet for video content (Migiro, 2018). Moreover, since 2011, digital platforms (including YouTube) have passed newspapers and television as a primary source of news (Pew Research Center, 2008, 2011), while cell phones between 2015 and 2017 surpassed all other digital-access device types (including personal computers) as the most common means for accessing information online (Bello-Bravo et al., 2021).

Despite the self-evident promise that YouTube affords, little remains known about which educational videos from academic (higher-education) institutions (HEIs)—specifically science animated videos posted on social and media platforms—are achieving impacts in terms of message delivery, receipt, and actionability. In terms of the framework for access described Ribot and Peluso (2003), although such videos have been made available, the question remains whether they are also (1) being used by recipients and (2) at the desired scale and range of recipients intended by the video's creators. This paper addresses these key information-delivery questions not only toward providing insights into creating accessible and impactful informal educational science animation videos online for users across the globe but also toward identifying metrics that HEIs can draw on to measure reach and impact.

Related research

The use of informal Internet-based ICT education videos on YouTube is not only providing a powerful means for transmitting information, education, and entertainment (Park, Naaman and Berger, 2016b; Platt et al., 2015), especially through the phenomenon of “Edutubers” (educational YouTubers) (López et al., 2020; Pattier, 2021a, 2021b, 2021c), but is also becoming more popular than conventional, studio-produced content (McRoberts et al., 2016; Tadbier and Shoufan, 2021). There is little doubt that educational animations on YouTube are achieving a much broader global reach even as specific impacts remains unclear (Claflin et al., 2021).

Although still in an emergent stage, higher education institutions (HEIs) are also paying significant attention to, and can sometimes adopt, YouTube as a means for sharing educational videos, whether as live-action or using animated media (Thelwall et al., 2012)—the latter, in part, because animations are not only generally more cost-effective to produce and update when necessary (Cantor et al., 2004; Eriksson and Eriksson, 2019; Fischer et al., 2005; Lowe, 2001a; Vinayagamoorthy et al., 2004) but also can communicate dynamic (scientific) ideas and processes in less abstract, more “digestible” ways (Bello-Bravo and Pittendrigh, 2018; Goff et al., 2017; Koch et al., 2010; Lowe, 2001b).

However, HEI use of digital online affordances for formal and informal education varies enormously by context depending on specific needs and strategies, including but not limited to overcoming technological access challenges (Zink et al., 2008), ensuring pedagogic effectiveness, flexibility, and adaptability (Delfosse et al., 2014; Ebner et al., 2020; Finger, 2014; Govindasamy, 2001), and operating cost-effectively (Klotz and Wright, 2017; Tay and Low, 2017; Wu and Huang, 2007) in both national, international, and internationalized contexts (Al-Azawei et al., 2016; Ebner et al., 2020; Govindasamy, 2001; Mahboobi, 2021). For informal education specifically, HEIs can also host otherwise independent entities that generate informal educational material as part of larger missions to intervene into local or global problems (MSU, 2020; SAWBO, 2021a). This study examines one such case in Scientific Animations Without Borders (SAWBO). Importantly, the scope of this study includes only a quantitative analysis of metrics around educational materials posted on SAWBO's YouTube channel; sociological variables around the relationship of SAWBO and its hosting HEI, while important, are beyond the scope of this research.

To quantitatively assess the reach of this latter type of informal education online, video platforms, like YouTube, offer analytics to measure the variables of channel and video awareness (i.e., view count and subscribers), consideration (i.e., watch-time), and action (e.g., likes, dislikes, comments, and shares). These metrics afford channel owners and researchers quantitative insights into social video viewing and sharing. With more than 2,500 studies to date (from 2007-2019) using YouTube data to draw general conclusions about individual videos on specific topics, these approaches now afford high-level statistical analyses into video/viewer interaction, predictions around video popularity, and external factors that affect how videos are being used and shared (c.f., Aggrawal and Arora, 2019; Bärtl, 2018; Chatzopoulou et al., 2010; Madden et al., 2013). While these studies, which have focused on larger aggregates of data, generally use view count (Chatzopoulou et al., 2010) as the main proxy for video popularity or quality, this is clearly a formal assumption in need of better validation. Wide distribution and a very publicly visible footprint can drive absolute view count without necessarily indicating popularity or quality; the name Pol Pot (or any other major historical figure associated with genocide), for example, is well-enough known to serve as an obvious example of this point, but he is hardly popular except possibly in some very niche demographics. Similarly, if view count only tallies the “click” on a video—which may suffice in a context of ad-driven models on the Internet (Kononova et al., 2020; Wang, 2020)—then total watch-time may better indicate popularity in terms of the user's engagement with the video (Wu, Rizoiu and Xie, 2018). These questions are by no means settled, and this study accepts view count as a proxy for popularity simply as a formal matter. This study complements such an approach by including other YouTube metrics in conjunction with view count to fill a knowledge gap around data on science-based informal educational animated videos on YouTube.

In general, access data on YouTube indicates what, where, and how frequently video access points occur with respect to time, geographical location (country or region), and ostensibly demographics (age and gender)—only ostensibly because no consistently reliable means exist to confirm or guarantee that the user-reported demographics correspond to the user's actual demographics. Nevertheless, by disclosing how and when informal educational ICT messages were accessed and viewed (and for how long) by recipients, an analysis of user patterns and trends at the channel-level, including the global reach of top videos, can help validate the relationship between user activity metrics and messages transmitted (e.g., Park et al., 2016b; Saurabh and Gautam, 2019). While confirming whether primary success metrics (i.e., view count and subscribers) correlate at the channel- and video-levels is key, it also exposes gaps in the data that further research (or different methods) could fill.

Aim of the study

This work is one of the very first scholarly studies to investigate the popularity of informal HEI education science animation videos on YouTube. Specifically, the study provides a time-based characterization of science animation videos offered by one US-based HEI's YouTube channel, Scientific Animations Without Borders (SAWBO). Established in 2010, and currently housed at Michigan State University in the United States, SAWBO has collaboratively researched, created, and freely distributed more than 1500 scientific animated videos translated into 240 + community's most comfortably spoken dialects (as of October 17, 2021). While hosted by an HEI, SAWBO independently produces its own content without institutional constraints. A crucial part of its mission involves delivering essential information in the core areas of agriculture, health, and women's empowerment to the widest demographic possible. At the time of this writing, videos on the SAWBO YouTube channel have more than 38 thousand subscribers (SAWBO, 2021b), more than 14 million views (SAWBO, 2021c), and are freely available for download, redistribution, and sharing (viewing) on mobile phones (social media, or other ICT devices). Specifically, this study

analyzes the SAWBO channel's user-activity through YouTube metrics of awareness, consideration, and action

validates channel- and video-level metric relationships to confirm the most predictive metric for maximum video access, reach, and impact, and

analyzes geographic and apparent demographic (age, gender, age x gender) data to tentatively characterize SAWBO's audience, reach, and user engagement with video content

identifies the most viewed SAWBO videos by language (English or non-English).

Methods

This quantitative study draws on analytic and channel-owner data from the SAWBO's YouTube channel. This channel was selected because access to its channel-owner data afforded a deeper analysis than publicly available data would allow (Saurabh and Gautam, 2019) and a more focused analysis of correlations (if any) between YouTube video metrics and user engagement.

Data collection

YouTube Analytics report data in terms of awareness (i.e., view count and subscribers), consideration (i.e., watch-time), and action (e.g., likes, dislikes, comments, and shares). Drawing on SAWBO's channel-owner content and activity data from 18 February 2011 to 9 October 2018 (https://purr.purdue.edu/publications/3907/1), collected data from YouTube Analytics was disaggregated into watch-time (e.g., watch-time, view count), traffic and interaction (including subscribers) reports, number of view count per video, number of subscribers, and number of likes, shares, and comments per video (see Table 1 for definitions). Videos were also classified by language using data from the main SAWBO site, given that YouTube does not track this information.

Table 1

YouTube user activity and engagement metrics and definitions.

Metric	Definition and Interpretation
Watch-Time	Amount of time (in minutes) that viewers have watched a video
View count	Number of times a video has been watched
Subscribers	Viewers that subscribed to the channel and were captivated by the channel's videos
Likes/Dislikes	Reflection of the emotional reaction of the viewer; viewer will either like or dislike the video
Comments	Demonstrates viewer's intention to interact with video creator via positive or negative feedback
Shares	How many times video content has been shared on social media; indicates that viewers watched the video and were also engaged to share it to YouTube and/or other sites

Data analysis

Descriptive statistics for each metric and an in-depth time-series analysis of collected data were performed to characterize trends for SAWBO videos on their YouTube channel. Change in view count, watch-time, likes, comments, shares, and subscribers over time were computed using the compound annual growth rate (CAGR), which captures year-to-year constant growth rate over a given period. Demographic information—including user-reported age, gender, age x gender, and geolocation—were collected at the channel-level. The top SAWBO channel videos were identified using a composite score based on equally weighted video metrics, namely: watch-time, view count, likes, and shares. Relationships between total view count, total watch-time, total likes, total comments, total shares, and total subscribers were calculated. Data analytics and visualization, including correlation analyses for metrics, were done using Tableau ver. 2019.1. Additional analyses (e.g., non-parametric statistical testing) were done using Real Statistics Resource Pack for Excel 365. See Supplemental tables 1–3 for detailed results. No analysis of comments was undertaken, as this would have involved qualitative methods outside the scope of this study.

Results

Objective 1: Analysis of SAWBO channel user-activity per YouTube metrics for awareness, consideration, and action

Overall, this study analyzed more than 470 videos created for English- (29.75%) and non-English (70.25%) speaking communities worldwide. Analysis in terms of YouTube metrics of awareness (i.e., view count and subscribers), consideration (i.e., watch-time), and action (e.g., likes, dislikes, comments, and shares) yielded a daily mean 856 view count, 2,042 min watched, 4.32 likes, 0.24 comments, 7.92 shares, and 3.72 subscribers, with maximums of 15,403 (view count), 58,662 min (watch-time), 135 (likes), 25 (comments), 102 (shares), and 97 (subscribers) [see Table 2].

Table 2

Descriptive statistics for SAWBO videos between 18 February 2011 to 9 October 2018.

Metrics		Sum	Daily Mean	Daily Min	Daily Max
View counts		2.39 M	856	0	15,403
Watch-Time (minutes)		5.70 M	2,042	0	58,662
Likes		12,075	4.32	0	135
Comments		673	0.24	0	25
Shares		22,111	7.92	0	102
Subscribers		10,396	3.72	0	97

M = million.

Objective 2: Validation of channel- and video-level metric relationships to confirm the most predictive metric for maximum video access, reach, and impact

A time-series analysis of the SAWBO channel covering the data period 2011–2018 revealed a statistically significant (p < 0.01) year-wise growth trend for select YouTube metrics using the Mann-Whitney U test (two-tailed, p < 0.05) (see Table 3 and Figure 1). Specifically, watch-time, likes, shares, and subscribers increased more than 100% over the past seven years since the channel's launch in 2011. Geographically, SAWBO videos were watched in more than 120 countries (Figure 2), with a majority of videos viewed in the United States (despite most of the videos having non-English titles) (data not shown). SAWBO videos were also most watched by people in Brazil, India, and Spain (data not shown), which are in the top 10 countries for YouTube viewers (ChannelMeter, 2019), and Mexico.

Table 3

SAWBO video compound annual growth rate (CAGR) between 18 February 2011 to 9 October 2018.

	2011	2018	2011–2018	CAGR (%)
Total View counts	40,020	844,570	2.37 M	66
Total Watch-Time (minutes)	17,362	2.25 M	5.7 M	125
Total Likes	85	5,368	12,034	100
Total Comments	19	136	661	39
Total Shares	55	9,822	22,089	137
Total Subscribers	41	4,711	10,385	120

M = million.

Figure 1

SAWBO channel event type total numbers by time within and between years. The plots of total view counts, total watched time (minutes), total likes, total comments, total shares, and total subscribers.

Figure 2

SAWBO's geographical reach (between 18 February 2011 to 9 October 2018). SAWBO's videos have been accessed by YouTube viewers from more than 120 countries. In green are given those countries where known views have occurred on the SAWBO YouTube channel. In red are given countries where no known use has occurred either through no actual views or due to no data being available/accessible. In multiple cases, islands are labeled green because they are part of countries where views have occurred.

Objective 3: Analysis of geographic and apparent demographic (age, gender, age x gender) data to tentatively characterize SAWBO's audience, reach, and user engagement with video content

A demographic profile of user-reported data for gender, age, gender x age was also conducted. Viewers aged 45–54, who were also the primary audience a year after SAWBO was launched, also had the highest accumulated number of views and watch-time (see Figure 3A-B). By 2018, SAWBO's audience had diversified and penetrated all age groups (from the youngest, age 13–17, group to the oldest, 65 + years) with the 25–36 age group having the most view count (30%) and watch-time (28%). Thus, the demographics pertaining to age became more inclusive over time. Consistent with other demographic analyses of YouTube (Blatberg, 2015), more men watched SAWBO's videos across all age groups (see Figure 3).

Figure 3

Viewer Demographics for SAWBO Channel. (A) View count (number of views) per year by age range. (B) View count (watched in minutes) per year by age range. (C) Percentage of view counts by age range and gender by year. (D) Percentage of watched minutes by age range and gender by year. For C and D, values above 0.1% are plotted on the graphs.

Objective 4: identification of most-viewed SAWBO videos by language (English or non-English)

By composite score (e.g., view count, watch-time, likes, shares, comments, and subscribers), the video Survival Gardening: Drip Irrigation—originally translated into Spanish and describing a technique for evenly watering entire crops—had the most total view count (662,570), highest total watch-time (2.36 million minutes), and most total likes (4,639), total shares (5,503), total comments (52), and total subscribers (3,052). Videos on charcoal water filtration and tuberculosis prevention, in Portuguese and English, were the second- and third-most viewed SAWBO videos (see Table 4).

Table 4

Metrics of most-viewed SAWBO videos between 18 February 2011 to 9 October 2018.

Totals	Video Title
	Survival Gardening: Drip Irrigation (Spanish)	TB Prevention (English)	Charcoal Water Filtration (Portuguese)
View counts	662,570	73,908	211,558
Watch-Time (Minutes)	2.37 M	117,204	334,511
Likes	4,639	186	277
Comments	52	26	20
Shares	5,503	667	528
Subscribers	3,052	80	106

M = million.

At the channel level overall, all metrics except for comments were found to have a statistically significant (p < 0.01) positive correlation with view count (see Figures 4 and 5). A statistically significant positive correlation was found between: (i) subscribers and view count (Figure 4); (ii) view count as compared with watch-time, likes, and shares (Figure 5A,B,D); and (iii) subscribers as compared to watch-time, likes, and shares (see Figure 6A,B,D). View counts and comments (Figure 5C) as well as subscribers and comments (Figure 6C) were not statistically significant as per a correlational analysis.

Figure 4

Correlation of view counts by subscription for SAWBO videos (R2 = 0.9797, F (1, 5) = 159.74, p < 0.01).

Figure 5

Correlation of view counts by (A) watched time in minutes (R2 = 0.9958, F (1, 5) = 1175.78, p < 0.0001), (B) likes (R2 = 0.9724, F (1, 5) = 176.38, p < 0.01), (C) comments (R2 = 0.5502, F (1, 5) = 6.11, p = 0.056; not significant), and (D) shares (R2 = 0.9830, F (1, 5) = 289.36, p < 0.01).

Figure 6

Correlation of subscribers by (A) watched time in minutes (R2 = 0.9866, F (1, 5) = 368.62, p < 0.01), (B) likes (R2 = 0.9990, F (1, 5) = 4912.41, p < 0.01), (C) comments (R2 = 0.5074, F (1, 5) = 5.15, p = 0.073; not significant), and (D) shares (R2 = 0.9973, F (1, 5) = 1838.97, p < 0.01).

Discussion & future research

Overall, the analysis of the relationship between channel user-activity and YouTube engagement metrics reaffirms other findings’ statistically significant correlations between YouTube popularity metrics (view count, likes, watch-time, subscriptions, and shares) and YouTube content (Chatzopoulou et al., 2010; Lopezosa et al., 2019). Similarly, the findings that more view count, watched, likes, and shares garnered more subscribers and that more view count garnered more watched, likes, shares, and subscribers also reaffirm other findings that correlate popularity metrics with a higher chance for subscribers (Berger et al., 2019; Langworthy, 2017; Lopezosa et al., 2019). For monetization and ad-driven contexts, these correlations with view count may already suffice as usable data for decision-making (Kononova et al., 2020, although see Drèze and Hussherr, 2003), but for efforts aimed at engaging particular demographics or assessing what content is most useful to those accessing it, considerable challenges to be faced by future research remain.

Content access (“Who Goes There?”)

With respect to who is accessing a given video content, Ribot and Peluso (2003) emphasize that a resource's mere availability does not yet provide its accessibility; in addition to availability, people must also be able to use that resource (whether because it is freely available to all, because they have enough money or social capital to be permitted to use it, or because they are not otherwise arbitrarily barred from its use due to some personal or social characteristic, such as gender, age, tribal or political affiliation, and so on). While every data point in this study by definition represents an instance of someone accessing [being able to use an available] video, determining the demographics for who actually accessed it presents considerable difficulty given the likelihood of mismatches between user's self-reported and actual demographics.

While no clear consensus as yet exists for the validation of user-reported data generally (c.f., Durmaz et al., 2020; Krause et al., 1999; Pole et al., 2006; Short et al., 2009), one of the more relevant factors for being able to trust user-reported data's accuracy—besides participants correctly understanding any questions asked, correctly recalling the actual answers to those questions, not leaving questions unanswered, and not participating in surveys multiple times—are situational aspects during the data solicitation that encourage or discourage respondent accuracy. These situational aspects include but are not limited to an assured anonymity of response, decreased or no fear of reprisals for answering accurately, and decreased pressure from social-desirability bias (Durmaz et al., 2020; Short et al., 2009), i.e., the tendency of respondents “to stretch the truth in an effort to make a good impression” (Martin and Nagao, 1989, p. 72). Moreover, although it is intuitive that computer-mediated solicitation of user-response data could decrease social desirability bias, the results are actually mixed (Booth-Kewley et al., 1992; Lautenschlager and Flaherty, 1990; Leichtmann and Nitsch, 2020). Lautenschlager and Flaherty (1990) found that social desirability bias increased when participants used computers to self-report compared to pen and paper, but Booth-Kewley et al. (1992) were unable replicate this finding.

Nevertheless, online platforms that afford or require profile creation (or the collection of demographic information generally) are hotbeds of user self-misrepresentation (Drouin et al., 2016; Steinel et al., 2010). Some of this misrepresentation is driven by social desirability bias on social media and other publicly visible digital platforms (Steinel et al., 2010), precisely because users can “stretch the truth in an effort to make a good impression” (Martin and Nagao, 1989, p. 72). But site users are also concerned about actual or imagined “reprisals” from unintended consequences of their online activity, e.g., spam, privacy violations, unwanted solicitations, or doxing (Davazdahemami et al., 2020; Hui et al., 2006; Kolotylo-Kulkarni et al., 2021; Trottier, 2020) as well as breaches or compromises of their anonymity, e.g., through cookies, malware, phishing, and online monitoring and tracking in general (Imani, 2021; Wakefield, 2004, 2013; Yu et al., 2020). These three situational elements—concerns about reprisals, breaches of anonymity, and increased social desirability bias—mark the very opposite of conditions that support validity in user self-reported data. Accordingly, this situation requires but also presents considerable technical and ethical challenges for future research that would seek to validate demographic conclusions based on user-reported data not liable to user-consented triangulation or other independent validation (Fedushko, 2019; Korobiichuk et al., 2017; Legewie and Nassauer, 2018).

Message reach and engagement (“It's Gone Viral!”)

Although the main results of this study confirm something intuitively obvious, these and the other related findings like them not only reaffirm a necessary prerequisite of availability for achieving accessibility generally but also open up pathways to notions like virality as an emergent variable for assessing a message's reach (Alhabash and McAlister, 2015). Importantly, while reframing virality, Alhabash and McAlister (2015) take care not to equate the term with popularity (also see Gunthert, 2009) or conflate the reach of a message with other terms used to measure or infer message effectiveness. For example, they cite Drèze and Hussherr's (2003) conclusion of a “deficient reliability” (Alhabash and McAlister, 2015, p. 2) for measuring message effectiveness using view count and click-through-rates; they also challenge whether inferences of user engagement from popularity metrics like YouTube's actually indicate message effectiveness (Tucker, 2011).

In their framing of virality, viral reach indicates “number of users who viewed the online message” (Alhabash and McAlister, 2015, p. 3); for HEI videos on YouTube, this would be view count (as the number of users who accessed a video). Moreover, because their framing of virality draws on motivation as the most relevant factor for users' interactions with digital media (Alhabash et al., 2013)—and entertainment as the most prominent motivation (Alhabash and McAlister, 2015; Cheng et al., 2008)—then their component of affective evaluation in virality reflects YouTube likes or dislikes but in principle could include watch-time (Chen et al., 2013; Zink et al., 2008), particularly in terms of the percentage of the entire video watched by a viewer. Indeed, while the total length of a video can influence whether it is fully watched or watched at all (Park, Naaman and Berger, 2016a; Yang et al., 2016), videos that are more liked and more positively commented have longer watch durations (Park et al., 2016a; Yang et al., 2016). Thus, percentage of video watched would provide a rather literal measure of viewer engagement with a given video (without disclosing anything as yet about the viewer's ultimate use of the video's content).

Content creators and engagement (“Don't Kill The Messenger”)

While a ready temptation exists to understand the popularity or virality of a video in terms of some affective evaluation in the above sense (Alhabash and McAlister, 2015), not only do those authors avoid this use of popularity entirely, they note how measurements of video popularity and its relation to virality, if any, are not at all well-established. Researchers have treated popularity and virality as identical in some studies, in others carefully differentiated them, and in still others operationalized one in terms of the other (Deza and Parikh, 2015; Goel et al., 2016; Gunthert, 2009; Weng et al., 2013). As one example of the ambiguity around these terms, in Berger et al. (2019)—which examines viewer affective evaluations of video content creators, not their popularity or virality—the data set included one British video content creator with 3.1 million views, 375,051 likes, and 6.5 million subscribers and a North American content creator with only 2.1 million views, 192,500 likes, but 23.2 million subscribers (p. 1762). Which of these data in isolation or as some aggregate score might associate with popularity remains wholly obscure.

Nevertheless, because demographic and presentational aspects of video content creators or presenters can play a role in affective evaluations of YouTube videos (Berger et al., 2019; Han, 2020; Han, Lappas and Sabnis, 2020), this opens the possibility of increased or decreased affective evaluations when using animated (not live-action) content presentation. For example, Berger et al. (2019) found that gender significantly affected view count but not likes (with videos by male creators garnering more views than female creators). Also, the speaking charisma of content creators also significantly affected subscriptions (with North American creators garnering more subscribers than British creators).

While testing hypotheses related to charismatic speech as a possible driver of affective evaluations of YouTube videos, Berger et al. (2019) acknowledges several limitations and caveats to the approach (differences of viewership above all), including the general difficulties of quantifying speech charisma in the first place (Michalsky and Niebuhr, 2019). Nevertheless, there may be a link between the disproportion of affective evaluations by viewers, accentual differences between content creators' North American and British speech, and the basic insight of increased messaging appeal for recipients when delivered in their most comfortably spoken dialect (Nonaka and Takeuchi, 1997; Rodríguez-Domenech et al., 2019). More broadly, given that engagement, learning gains, and solution uptake from animated educational contents are at least equal to, if not greater than, live-action teaching or video content (Bello-Bravo, 2020; Bello-Bravo et al., 2020; Bello-Bravo et al., 2017a, Bello-Bravo et al., 2017b; Smith et al., 2012), future research might compare the affective evaluations for YouTube's popularity metrics between animated, live-action, and combined animated/live-action channel content for individual video content creators (e.g., the Game Grumps, Markeplier, the Yogscast), whether self- or fan-created.

Moreover, educational animations like SAWBO videos have demonstrated these increased metrics for engagement, learning gains, and solution uptake regardless of the demographic variables of age, gender, socioeconomic status, educational or technological literacy, and geographic location (Bello-Bravo and Pittendrigh, 2018; Bello-Bravo et al., 2018; Mocumbe, 2016; Payumo et al., 2021). Such a capacity may hold a promise for overcoming the type of gender disparity for subscriptions noted by Berger et al. (2019). Additionally, these empirically demonstrated effects for motivating solution uptake and the actual use of the video's educational content in local communities around the world may ground at least to a certain extent the assumption that engagement with such videos on online platforms can indeed generate similar engagement in the world itself offline.

In sum, there is little doubt that YouTube is perceived by all kinds of organizations—including institutions of higher education—as a potential channel for delivering information, even after logistic, pedagogic, administrative, and geospatial aspects are factored in. YouTube undeniably represents a platform that academic institutions can use to bolster their formal and informal educational and outreach programs, particularly when communicating research and innovations that are of global importance (Asio and Khorasani, 2015).

Conclusion

While the first two objectives of this study, consistent with other research, generally affirm the analytical use and statistical correlations of YouTube's metrics for awareness, consideration, and action, the findings also sketch out the borders or limits for the usefulness of those metrics at the channel level for measuring the reach and impact of science animated videos distributed by higher education institutions (HEIs), especially around user demographics (objective 3). That is, while HEIs may be able to more reliably control any needed or wanted demographic analyses through self-hosted formal educational platforms, the current data quality afforded by YouTube, in conjunction with users' likely inclination to perceive site surveillance as a discouragement for disclosing accurate demographic information, effectively preclude such analysis in this informal setting. This generates an undesirable uncertainty around determining both who is being reached and whether or not an intended demographic is being reached.

Nevertheless, the data that YouTube does provide can be more cannily used to afford deeper insights into basic questions of engagement (for example, using the percentage of video watched in conjunction with affective evaluations such as like an subscribe). Or again, with regard to objectives 3 and 4 (demographic analysis and most-watched videos), a better operationalization of what the action “like” or “dislike” means through a cultural context that acknowledges and can account, for example, for statistically significantly decreased subscriptions for female (or potentially “foreign”) content creators would be a crucial step for future research. On this point, the demonstrated capacity of animation to avoid these kinds of cultural effects could motivate research to compare content creator's animated and live-action content, as a possible research angle to uncover novel ways to increase the likelihood of a content's virality.

Declarations

Author contribution statement

Julia Bello-Bravo: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data; Wrote the paper.

Jane Payumo: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Wrote the paper.

Barry Robert Pittendrigh: Conceived and designed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data; Wrote the paper.

Funding statement

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Data availability statement

Data associated with this study has been deposited at Purdue University Research Repository (PURR): https://purr.purdue.edu/publications/3907/1

Declaration of interests statement

The authors declare no conflict of interest.

Additional information

No additional information is available for this paper.