Investigating Learner's Online Learning Behavioural Changes during the COVID-19 Pandemic.

The objective of this study is to understand the pandemic's impact on online learning behaviour based on learners' comments collected from online video tutorials hosted on YouTube. The topic modelling approach is employed to uncover the changes in topic prevalence during the pre- and post-pandemic timeframe to infer learning behaviour. Ten topics were uncovered, and each exhibited a varying degree of changes over time. Overall, the study identified two learning behavioural changes, (1) learners were more active in learning relevant skillsets through sharing their experiences; and (2) learners had altered their help-seeking behaviour to aid in their learning. 84th Annual Meeting of the Association for Information Science & Technology | Oct. 29 – Nov. 3, 2021 | Salt Lake City, UT. Author(s) retain copyright, but ASIS&T receives an exclusive publication license.

INTRODUCTION

On March 12, 2020, the World Health Organization declared COVID‐19 as a global pandemic after assessing the severity of the virus (World Health Organization, 2020). Many countries have taken unprecedented measures to lockdown the cities, including the closure of schools to curb the virus transmission. As a result, learning has changed drastically, seeing a sharp increase in online learning (Shah, 2020; YouTube, n.d.), whereby learners access the learning content remotely on a digital platform such as massive open online courses and video tutorial hosting sites. Amid the high participation rate, there are concerns about the pandemic's impact on online learning (Mheidly et al., 2020). Studies have since focused on areas, for example, learners’ engagement, satisfaction, and perception of online learning (Alawamleh et al., 2020; Baber, 2020). There are also indications to suggest that the pandemic could affect behavioural changes in online activities (Popa et al., 2020; Rosa et al., 2020).

To date, there is limited study on the pandemic's impact on learners engaging in online video tutorials, in particular, on the use of learners’ online comments to uncover changes in learning behaviour. Posting online comments allows learners to share their learning experiences publicly or even seek guidance or clarifications when they encounter learning difficulties. Past studies have shown that it is valuable to analyse comments to understand commenters’ concerns or satisfaction (Chen et al., 2020; He et al., 2020; Lee et al., 2017), and feasible to use the computational technique to analyse user behaviour in an online environment (Kim & Cho, 2020). Thus, this study aims to address the current gap by proposing a topic modelling approach to analyse learners’ online comments from the content hosting site, YouTube, to understand learners respond to online learning before and after the declaration of the pandemic.

METHOD

According to a report by World Economic Forum, many people have picked up coding as a new skill or reskill themselves as technology jobs were seen to provide better job security amid the COVID‐19 pandemic (Marchant, 2021). As such, the dataset was built from a popular YouTube channel, the freeCodeCamp, that curated coding‐related video tutorials and have a large number of subscribers. A total of 52,431 comments were collected from 24 video tutorials (M = 2,185, SD = 3,437) that had more than a million views and published before 2019. Structural topic modelling (STM) was used for analysis as it had the advantage to incorporates documents' metadata such as date into the model in the form of covariates to estimate the covariates' moderation effect on the topics (Roberts et al., 2013).

First, the comments were filtered to focus on the seven months before and after the pandemic announcement to assess for potential changes in the two periods. Words were extracted from the comments and pre‐processed by removing stopwords, high‐occurrence words, numbers, punctuations, and followed by auto‐correction and lemmatisation of the remaining terms. Next, STM was used to uncover latent topics and compute the topic prevalence. Before building the STM model, the number of topics was evaluated using three metrics: held‐out likelihood estimation, semantic coherence, and residuals (Naab & Sehl, 2016; Roberts et al., 2013). The STM model was then fine‐tuned to obtain a good model fit determined by exclusivity and semantic coherence (Naab & Sehl, 2016). Topics generated were manually assigned with meaningful and representable labels to describe the topics based on the uncovered keywords and their associated comments.

RESULTS

The STM model uncovered ten topics and broadly grouped into two distinct categories: text‐based topics and coding‐based topics. Text‐based topics consist of descriptive comments, while coding‐based topics comprise primarily programming languages. For example, Topic 2 (free, exam, pass), categorised under text‐based topics, reflected how learners engaged in the learning activities. Similarly, in Topic 6 (work, error, create), learners posted coding questions and codes related to technical errors, and thus categorised as coding‐based topics. The text‐based topics comprised Topic 2, 3, 8 and 10, were associated with sharing learning practices, sharing compliments, sharing general feedback, and sharing specific feedback on video content, respectively. The remaining six topics were coding related comments that constituted about 50.7% of the overall expected topic proportion. These topics included Topic 1 (sharing learning achievement), Topic 4 (sharing clarifications), Topic 5 (sharing coding errors), Topic 6 (sharing technical errors), Topic 7 (sharing coding queries), and Topic 9 (sharing programming codes).

The combined topic prevalence plot in Figure 1 showed that nine topics converged towards a similar topic proportion during T , suggesting a shift in learning behaviours. Meanwhile, Topic 3 remains relatively high and stable throughout T and T compared to other topics. Figure 2 showed an example of the detailed topic prevalence plots for Topic 2 and 6 that exhibited a sharp rise and significant drop in the topic proportion leading towards T , respectively.

FIGURE 1

Combined topic prevalence

FIGURE 2

Topic prevalence over time for Topic 2 and 6

DISCUSSION AND CONCLUSION

This study presents a preliminary analysis on the use of topic modelling to study the pandemic's impact on learners engaging in online video tutorials and infer learning behaviour changes. Latent topics were first identified by the STM model, followed by computing the topics prevalence over the pre‐ and post‐pandemic timeframe.

The topic prevalence plots showed that there were differences in learning behaviour between the pre‐ and post‐pandemic. The convergence of the nine topics and the high fluctuation of the topics observed during T reflected the pandemic's volatility and effects. This phenomenon could be attributed to the different lockdown periods among the various countries during T and a higher take‐up and completion rate of learning (Hirsch, 2020; YouTube, n.d.). In particular, Topic 2 and 6 exhibited the most significant changes in topic prevalence. Topic 2 offered a perspective that active learning occurred during T when more learners shared their learning experiences during the lockdown period. The drop in Topic 6 suggested a possible change in the type of questions posted during T2. In sum, these changes highlighted events such as the pandemic could alter learners’ concerns and learning focus. Thus, it is important for educators and content creators to quickly identify these changes to better address learners’ needs and support them.

This preliminary study may be limited by the homonyms used among the comments. Thus, it required additional effort to ensure that the keywords generated were interpreted correctly against its associated comments. To address this, we plan to explore the use of semantic‐based topic modelling to improve topic identification. In addition, this study focuses on coding‐related learning and could not be generalised to other types of courses. Hence, we also plan to investigate other learning domains (e.g., learning languages). Against the backdrop of interrupted schooling from the pandemic where online learning will become more common and increasingly important, this study is critical and contributes to better understanding of how learners responded to online learning during the pandemic.