Evaluating students' engagement with an online learning environment during and after COVID-19 related school closures: A survival analysis approach.

BACKGROUND: Due to the COVID-19 pandemic schools all over the world were closed and thereby students had to be instructed from distance. Consequently, the use of online learning environments for online distance learning increased massively. However, the perseverance of using online learning environments during and after school closures remains to be investigated.
METHOD: We examined German students' (n ≈ 300,000 students; ≈ 18 million computed problem sets) engagement in an online learning environment for mathematics by means of survival analysis.
RESULTS: We observed that the total number of students who registered increased considerably during and after school closures compared to the previous three years. Importantly, however, the proportion of students engaged also decreased more rapidly over time.
CONCLUSION: The application of survival analysis provided valuable insights into students' engagement in online learning - or conversely students' increased dropout rates - over time. Its application to educational settings allows to address a broader range of questions on students' engagement in online learning environments in the future.

Introduction

In March 2020, COVID-19 related school closures required alternative approaches of distance learning to instruct students. One such approach was the use of online learning environments, which allow teachers to assign digital learning material online to their students and from distance [1,2]. Mathematical problem sets computed by students can be scored automatically by appropriate software, and students as well as teachers receive feedback on whether problems were solved correctly [1], [2], [3]. This online teacher-student interaction provided the possibility evaluate students’ learning progress on these assignments.

The sudden school closures in March 2020 pushed many schools – teachers and their classes – to use such online learning environments [1,2,[4], [5], [6]]. For example, more than 67,000 students registered with the online learning environment Bettermarks in Germany during the first two weeks of the first school closure (i.e., between March 15th and April 1st, 2020). Similar increases in users due to COVID-19 related school closures were reported for other online learning environments in other countries (in terms of the numbers of users and computed problem sets, respectively [4], [5], [6]).

However, it remains to be investigated for how long teachers and students persevered in using these online learning environments during COVID-19 related school closures and whether the increased use of online learning environments persisted and will persist as a sustainable amendment to traditional learning approaches. To address this question, we investigated students’ engagement with the online learning environment Bettermarks for mathematics by means of survival analysis.

Survival analysis is usually applied in medical studies in order to compare different treatments against each other across patients on the chance of surviving over time [7]. However, survival analysis can also be applied to other contexts. For instance, recent studies used survival analysis to evaluate students’ engagement within online learning environments [8], [9], [10], [11], [12], [13], [14]. In this study, we applied this approach to evaluate students’ engagement – and, conversely, dropout rates – over time for different time periods as observed for the online learning environment Bettermarks.

With a first survival analysis, we evaluated data from students who registered with the Bettermarks online learning environment for mathematics within the first two weeks (i.e., March 15th – April 1st, 2020) of the first COVID-19 related period of school closures, and compared these students to another sample of students who registered within the same time window over the previous three years (i.e., March 15th – April 1st, 2017 to 2019). We then examined the total number of students as well as the proportion of students over time who actively engaged with Bettermarks during COVID-19 related school closures as compared to the same time period over the previous three years.

In particular, we compared students’ survival probability against that of students over the previous three years, as students’ survival probability may have varied between years due to other factors than school closures. If, however, an effect of school closures on students’ engagement with online learning environments is observable, then students who registered during periods of school closures should follow a different survival curve than students in previous years without school closures. Thus, comparison against previous three years served as a kind of benchmark - similar to other recent studies in educational research [15,16].

With a second survival analysis, we sought to replicate the first analysis with data from the second period of COVID-19 related school closures in Germany which lasted from the beginning of January until end of February 2021. As for the first period of school closures, we included students who registered within the first two weeks of schools being closed (i.e., January 1st – January 15th) to evaluate the robustness of a potential effect of school closures.

Finally, we explored potential aftereffects of COVID-19 related school closures, as teachers and students may have adopted to new ways of online learning/teaching even after schools re-opened again. In particular, we evaluated whether the result pattern observed in the first survival analysis was similar after schools opened again. Therefore, we evaluated the survival curve of another cohort of students who registered with Bettermarks during summer holidays in Germany (i.e., between July 1st, and September 10th, 2017, 2018, 2019, and 2020).1

Material and methods

Online learning environment

We analyzed data from the curricular based online learning environment Bettermarks for classes 4–10. This online learning environment is used by public and private schools as well as by vocational and academic track school types in Germany. Currently, as of the 2021–2022 school year, students in five states in Germany have free access to Bettermarks. In addition, Bettermarks is used in the Netherlands and Uruguay [2].

Teachers interact with their students by assigning problem sets to them. Students can solve these assigned problem sets and get immediate feedback on their performance. Teachers also receive feedback on students’ performance on the assigned problem sets. This teacher-student interaction allows for the use of Bettermarks for homework assignments as well as for solving problems within the classroom. In addition, students can also choose their own problem sets when they want to study on their own. Just as they do when learning with paper and pencil, students may voluntarily decide to stop studying with Bettermarks. Even when their teachers assigned problem sets to them, students may decide not to do their homework assignments. As regards teachers, they may decide not to assign problem sets to their students anymore and use another format to teach mathematics to their students. Importantly, all data available from Bettermarks are entirely anonymous and thus, no personal information (such as gender or age) from students or teachers can be identified.

Student data

Two criteria were applied to data from Bettermarks in order to be included in this study. First, we considered students who registered with Bettermarks within three specific time windows, each of which was considered for four years: Students who registered with Bettermarks i) between March 15th and April 1st in 2017, 2018, 2019, and 2020 as well as ii) between January 1st and January 15thth in 2018, 2019, 2020, and 2021, and iii) between July 1st and September 10th in 2017, 2018, 2019, and 2020 were included. Time periods i) and ii) represent times of school closures in Germany for years 2020 and 2021, respectively. The third time window was chosen to investigate aftereffects of the first COVID-19 related school closures in times with no school closure.

Second, students’ data from first registration and the following 365 days were included for the first and third time window. For the second time window, only data from the registration time period of a specific year until July 1st of that year were included. Importantly, this inclusion criterion allowed us to investigate students’ engagement within the first year (or first half year in the second survival analysis) from registration on only. Students who registered in earlier years may have stopped within the first year and then used the system again after more than one year. These students were only considered during the first year after their registration to investigate students’ engagement within the first year after registration (first half year for students who registered in the time window January 1st until January 10th). With these inclusion criteria applied, we included data from more than 300,000 students who computer more than 18 million problem sets.

Survival analysis

The survival function S(t) defines the probability of surviving over time. We re-labelled this function as P(Still Active) to apply survival analysis for engagement in the online learning environment Bettermarks. Consequently, we defined ‘survival’ as students who were still actively using the environment and ‘survival time’ as the number of active days passed since initial registration, with the timepoint 0 indicating the registration date of a student. Finally, students’ most recent last activity was defined as their dropout day (‘death’). This binary coding scheme of students’ being still active (coded as 1) and not active anymore (coded as 0) served as the basis for the survival analysis.

The interpretation of survival analysis in education is similar to survival analysis in medical research, with students still engaged with the system at a specific timepoint defined as active students (similar to an alive patient) and students’ final activity as their disengagement (similar to patients’ death in medical scenarios). As such, the application of survival analysis in educational settings allowed us to evaluate students’ engagement with the online learning environments over time.

To evaluate the influence of COVID-19 related school closures on students engagement in the online learning environment, we conducted a survival analysis using the Kaplan-Meier method, computed with the survival package [17] in R and RStudio [18,19]. The ‘Survfit’ function of the Kaplan-Meier method generated the survival curve with the survival function P(Still Active) as the dependent variable and days active considered as the independent variable. We tested the effect of each time period on students’ survival time using the Cox proportional hazard model with a random effect for each student using the ‘coxme’ package [20] in R. We quantified differences using the Cox proportional hazard model between time windows for the first 100 days after registration (note that Figs. 1 and 3 display the first 365 days after registration; Fig. 2 displays ≈180 days after registration).

Fig. 1

Survival curves indicating the probability of still active students (y-axis) as a function of active days (x-axis) for four consecutive cohorts of students (upper panel) as well as the total number of students still active (lower panel). As can be read from the lower panel, between March 15th and April 1st, 2020, the total number of students increased remarkably. However, these students were not active on Bettermarks for as long as students in the previous three cohorts. Day zero indicates the day of registration. Students’ final activity was defined as the day of leaving the online learning environment. We only considered students’ activity up until one year after their registration. In the upper panel, colored lines indicate the average probability of still using the online learning environment (survival curves). Shaded areas indicate 1 standard error of the mean (SEM). Please note that shaded areas for the purple curve cannot be identified due to the rather small SEM.

Fig. 3

Survival curves for aftereffects of school closures (upper panel) and total numbers of students registered (lower panel), compared to the same periods in the previous three years. The survival analysis included students who registered in the summer between July 1st and September 10th, in each respective year. Numbers in the lower panel suggest that considerably more students signed up with the online learning environment during summer holidays 2020. However, again engagement decreased faster over time for these students, as compared to students who registered within the same period in the previous three years. In the upper panel, colored survival curves indicate the average probability of still engaging with the online learning environment. Shaded areas reflect 1 SEM, but can hardly be seen.

Fig. 2

Survival curves for the period of second school closures (upper panel) and total numbers of students registered (lower panel), compared to the same periods in the previous three years. As for the first period of school closures, numbers in the lower panel reflect that more students registered during school closures. However, these students again disengaged significantly faster over time as compared to the previous three years. In the upper panel, colored lines indicate the average probability of students still using the online learning environment for the different years. Shaded areas reflect 1 SEM.

Results

The survival curves reflecting students’ probability to stay engaged with the online learning environment Bettermarks are depicted in Figs. 1 to 3 . Results are significant for all three survival analyses: considerably more students signed up (a) during the first period of school closures, (b) during the second period of school closures, and (c) after school closures, as compared to the same time periods in the previous three years, respectively. However, we also observed that the proportion of students who engaged with the learning environment decreased faster over time for those students who signed up during school closures, as compared to the same time window in previous years.

In the following sections, we report results of each of the three survival analyses, together with results from the Cox mixed-effect models. Importantly, the difference between survival curves during and after school closures, when compared to the previous years, were significantly larger than differences between survival curves for each of the previous years (these comparisons served as controls).

Times of school closures

Registration dates: March 15th – April 1st

The Cox mixed-effect model indicated a significant difference between survival in the time window 2020 (i.e., students who registered during the first two weeks after schools closed) as compared to each of the three previous years (p < .001). Closer inspection of the survival curves revealed that the median survival time for students who registered in 2020 was 69 active days (95% CL: 68–69), compared to 257 active days (95% CL: 251–264) for students who registered in 2019, 306 active days (95% CL: 297–311) for students who registered in 2018, and 339 (95% CL: 399–399) for students who registered in 2017.

Registration dates: January 1st – January 15th

Results on the second school closure (in 2021) replicated the pattern of the first period of school closure. The Cox mixed-effect model revealed a significant effect (p < .001) for the independent variable time window 2021 indicating a difference between the survival curve for the year 2021 as compared to each of the other three previous years. Evaluating the survival curves revealed that the median survival time for students who registered in 2021 was 88 active days (95% CL: 85–89), compared to 119 active days (95% CL: 117–120) for students who registered in 2020, 122 active days (95% CL: 118–130) for students who registered in 2019, and 144 (95% CL: 140–150) for students who registered in 2018.

Aftereffects of school closures (Registration dates: July 1st – September 10th)

As in the first Cox mixed-effect model, the independent variable time window 2020 was significantly different (p < .001) from each of the other three previous time windows and indicated a difference between the survival curve for the year 2020 as compared to each of the other three previous years. Evaluating the survival curves indicated that the median survival time for students who registered in the summer of 2020 was 246 active days (95% CL: 246–247), compared to 263 active days (95% CL: 262–263) for students who registered in the summer of 2019, 259 active days (95% CL: 259–260) for students who registered in the summer of 2018, and 268 (95% CL: 266–270) for students who registered in the summer of 2017.

Discussion

In this study, we employed a survival analysis approach (following the Kaplan-Meier method) to evaluate the perseverance of students’ engagement in an online learning environment for mathematics during and after COVID-19 related school closures. The survival analysis revealed that the proportion of students who stayed active decreased more strongly when students registered during (i.e., during the first and second COVID-19 related school closures) as well as after school closures (in summer 2020), as compared to the same time windows in the previous three years. This pattern was observed even though considerably more students registered in total since the first COVID-19 related school closures, as compared to the same time windows in the previous three years.

These results highlight the importance of reporting student engagement in online learning environments not only in total numbers within a given period, but also considering changes in their engagement over time. While the total number of students is an important indicator of how many students are active at a certain time point (or within a certain period), proportional changes in engagement (e.g., since registration) are an important indicator of how persistent students use an online learning environment. As such, our results substantiate benefits of applying survival analysis in educational settings to gain deeper insights into the nature of students’ engagement with a learning environment over time. This adds an important aspect – the perseverance of using online learning environments over time – to the discussion about the actual use and potential of online learning environments for formal education.

Considering the first period of school closures (March to April 2020), the survival analysis indicated that about 75% of students disengaged from using the online learning environment within the first 100 days after registration. This percentage was considerably lower for the previous three years where only about 25% of the students disengaged within the first 100 days after registration. This pattern of results replicated for the second period of school closures (January to February 2021), with about 60% of students who disengaged from using the system after 100 days, compared to about 30% in the previous three years. Finally, we observed a similar result pattern for the analysis for the cohort who registered during summer holidays (July to September). However, this analysis revealed the largest difference between survival curves after 200 days, with about 35% disengaged students who registered in 2020 after 200 days, compared to 25% of disengaged students after 200 days in the previous three years.

In general, these findings are in line with findings of previous studies on dropout rates from online courses before the outbreak of the COVID-19 pandemic [[11], [12], [13],[21], [22], [23], [25], [26], [27], [28], [29], [30]]. However, these studies primarily considered students who studied using online learning environments on their own and not within a class-context (i.e., without a student-teacher interaction). As such our results add to these previous findings by indicating that even if teachers are involved in the use of online learning environments, dropout rates over time remain considerably high, especially for those who registered during or after school closures. Yet, future research may investigate whether differences between dropout rates exist between online learning environments which incorporate a teacher-student interaction, compared to others without teacher-student interactions.

In sum, our results highlight two critical challenges which online learning environments currently face and may be considered in future research: (i) how to keep students engaged over time (i.e., increase students’ perseverance)? and (ii) why some students (e.g., here, those registered during or after COVID-19 related school closures) seem to be more susceptible to disengage from using online learning environment than others? While the current study was not designed to answer these questions in particular, it seems sensible to assume that many students may have disengaged after schools re-opened again, as their teachers switched back to using more formal teaching formats. As this was a gradual process, which means it happened at different time points, in different states/counties in Germany, this may, at least partly, explain the faster drop in engagement as compared to previous years. However, this may only be part of the story as a steeper decrease of student engagement was also observed for students who registered during the summer holidays in 2020.

Another possible explanation may be that proportion of students (and/or their teachers), who actually want to engage with the system for a longer time period, decreases with increasing registrations. Conversely, the number of indecisive students (or students with indecisive teachers) may increase with increasing number of students. This effect (the more students/teachers registered, the larger the share of the unmotivated students/teachers for participation) was not only observed for students who registered during and after COVID-19 related school closures, but also for those who registered in the previous years.

This indicates that the observed effect is not caused by COVID-19 related school closures per se, but by several other factors, e.g., motivational factors, performance effects, and assignments policies. More research is needed to answer the question of how to keep students engaged in online learning environments by better understanding what factors influence their engagement/disengagement. In the reminder of the discussion, we will discuss avenues for future research which may address these questions more specifically by means of survival analysis, not least in order to substantiate the potential of this approach. Finally, we consider limitations of the current study.

Avenues for future research

The survival analysis approach applied here may be applied to address further questions on the sources of variance in students’ perseverance, which are likely to consist of several motivational aspects. Whenever students have to do something (as opposed to voluntarily doing something), primary motivation may suffer [31,32]. It is well known, that secondary motivation (“you must do this now in order to keep going to learn”) decreases primary motivation (“I want to do this”). So it appears that the way online-learning systems are introduced (“you must…” versus “we have the opportunity to…”) may play a larger role in predicting perseverance (and hence, success) in the use of online learning environments.

In addition, one may consider investigating the influence of students’ performance on their probability to stay engaged within online learning environments. Previous research from cognitive psychology indicated that task performance affects voluntary task choices [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44]. Interestingly, some of these studies suggested that error commissions led to a higher probability of disengaging [33,34,37,38]. However, others suggested that not too few and not too many errors, but rather 15% or errors, are optimal for learning [45,46] and to stay engaged [47], [48], [49]. Thus, it may be that a sweet spot of failure makes students learn best, and that the areas around this spot renders students prone to disengage from online learning environments.

Transferring these rather basic findings to the applied field of education – online as well as off-line – would be desirable to better understand why students disengage in general. This may be investigated by measuring students’ performance within the first few weeks after registration and then investigating survival curves for different cohorts of students with different average error rates.

In addition to aspects of student performance, another potential influence on students’ engagement is teacher behavior. It may be that the assignment policy of teachers affects students’ engagement with online learning environments. For example, some teachers infrequently assign a large bulk of problem sets to their students, while other teachers frequently assign smaller chunks of problem sets to their students. A large body of literature on the so-called spacing effect suggests that students seem to learn better when being assigned small chunks frequently [50], [51], [52], [53]. Testing whether these results – again mostly drawn from more or less controlled laboratory settings – can be observed in settings of real education (i.e., online learning environments), might inform teachers on how to assign problem sets to students to keep them engaged. This may be tested by asking whether different assignment policies within the online learning environment affect survival curves.

Potential limitations of the present study

There are aspects to this study that need to be considered when interpreting the results. First, there may be cohort effects: Students who registered before the COVID-19 pandemic may have had teachers enthusiastic about online learning possibilities long before the first lockdown in March 2020. In contrast, at least some students who registered during or after school closures, may have had teachers who preferred more traditional face-to-face, non-digital teaching approaches and who primarily switched to the online learning environment because of distance learning due to school closures. With the re-opening of schools these teachers may have disengaged from using online learning environments and so did their students.

Second, there is the question of generalizability of the present results to other online learning environments. Before drawing strong conclusions, our results should be replicated with data collected in other online learning environments. However, the online environment investigated here is globally among the most widely used online learning environments for mathematics. This even allows future research comparing students from Germany, the Netherlands, and Uruguay. Such cross-cultural research would reveal the generalizability our results from Germany.

Another point to consider is that the survival analysis we employed did not account for how active students were (in terms of number of problem sets solved or number of days active) between their registration and disengagement. Thus, the survival analyses reported here are limited to the first and last engagement of students in terms of accessing the online learning environment and do not consider the degree of engagement within this period of time. A more fine-grained analysis is certainly desirable.

Conclusion

The results of this study highlight the applicability of survival analysis for data collected with online learning environments. Results show – very clearly using a single figure – how many students remain active learners for how long. As such, survival analysis is a useful tool to investigate students’ engagement – or conversely, students’ dropout rates in learning environments – over time.

This study found that, whereas the total number of students using an online learning environment increased considerably during and after school closures, the proportion of students’ engagement decreased faster over time. Therefore, user numbers are not the only aspect to consider when evaluating the influence of the COVID-19 pandemic on the use of online learning environments. Students’ perseverance, i.e., their engagement over time, is also an important variable to consider. Future research must specify factors which contribute to more persistent engagement – and lower drop-out rates – in online and off-line learning. The study of online learning environments may benefit off-line learning just as well.

Ethical Statement

In this study, we are reporting a retrospective study of archived data which was fully anonymized before data analysis. Thus, it is not possible to track the data back to any software user.

Financial disclosure

The author did not receive funding for this article.

Declaration of Competing Interest

There are no known conflicts of interest associated with this publication.