Predict or describe? How learning analytics dashboard design influences motivation and statistics anxiety in an online statistics course.

Based on the achievement goal theory, this experimental study explored the influence of predictive and descriptive learning analytics dashboards on graduate students' motivation and statistics anxiety in an online graduate-level statistics course. Participants were randomly assigned into one of three groups: (a) predictive dashboard, (b) descriptive dashboard, or (c) control (i.e., no dashboard). Measures of motivation and statistical anxiety were collected in the beginning and the end of the semester via the Motivated Strategies for Learning Questionnaire and Statistical Anxiety Rating Scale. Individual semi-structured interviews were used to understand learners' perceptions of the course and whether the use of the dashboards influenced the meaning of their learning experiences. Results indicate that, compared to the control group, the predictive dashboard significantly reduced learners' interpretation anxiety and had an effect on intrinsic goal orientation that depended on learners' lower or higher initial levels of intrinsic goal orientation. In comparison to the control group, both predictive and descriptive dashboards reduced worth of anxiety (negative attitudes towards statistics) for learners who started the course with higher levels of worth anxiety. Thematic analysis revealed that learners who adopted a more performance-avoidance goal orientation approach demonstrated higher levels of anxiety regardless of the dashboard used. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s11423-021-09998-z. © Association for Educational Communications and Technology 2021.

Introduction

A large number of undergraduate and graduate programs include statistics courses as part of their core curriculum. Unfortunately, statistics is perceived as a difficult subject by many learners (DeVaney, 2016; Hanna et al., 2008) and, as a result, many learners develop negative feelings towards statistics courses, statistics content, and even statistics instructors. Not surprisingly, statistics anxiety has been explored in numerous studies as part of a broader conceptualization that involves math anxiety, self-efficacy, motivation, and goal orientation (Hsu et al., 2009; Onwuegbuzie, 2004; Zeidner, 1991). Importantly, statistics anxiety may be compounded in online learning environments due to lack of immediate feedback from instructors and the requirements for great levels of metacognitive skills as well as proficiency in technology, especially when statistical software is used (Hsu et al., 2009; Zimmerman, 2008). This is a critical issue because online learning continues to grow and has dramatically expanded as a function of the public health crisis created by the COVID-19 virus. If most academic leaders (66%) already considered online learning as an essential part of their long-term strategies (Allen & Seaman, 2014), this focus is likely to become even more prominent now that many educational institutions have developed their own structures to offer online instruction.

This study investigates how to better support learners’ affective and cognitive outcomes in online learning environments by using an important affordance of online contexts: learning analytics. More specifically, we designed and tested two learning analytics dashboards – predictive self-referenced vs. descriptive norm-referenced (Biesinger & Crippen, 2010; Koenka & Anderman, 2019) – to explore how different design and pedagogical features influence learners’ motivation, statistics anxiety, and learning performance in an introductory online statistics course. In this study, learning analytics dashboards are presented as visual displays of customized feedback on learning performance and information about the course timeline (Park & Jo, 2015; Tan et al., 2017). Despite the increasing interest in the potential of learning analytics to support learners’ reflection, behavioral changes, motivation, and learning performance (Gašević et al., 2015; Park & Jo, 2019), empirical studies investigating the influence of learning analytics dashboards on learners’ cognitive and affective outcomes are still limited (Bodily & Verbert, 2017; Jivet et al., 2018), particularly in authentic settings such as online statistics courses and that are based on an experimental design. This study represents a step to fill this void.

Theoretical background

Statistics anxiety

Statistics anxiety can be described as the anxiety learners experience when dealing with statistics “in any form and at any level” (Onwuegbuzie, 2004). Statistics anxiety is reflected by the mental disorganization, worry, and tension that learners feel when they have to work with statistics content or problems (Zeidner, 1990). Statistics anxiety usually develops from situational, dispositional, or environmental antecedents (Onwuegbuzie & Wilson, 2003).

Statistics anxiety is a multidimensional construct that differentiates between test- and content-related anxiety (Cruise et al., 1985; Onwuegbuzie, 1997; Zeidner, 1991). For example, Cruise et al. (1985) described six components of statistics anxiety: (a) worth of statistics (perceived usefulness of statistics, negative attitude towards statistics); (b) interpretation anxiety (anxiety when interpreting statistical results); (c) test and class anxiety (anxiety experienced when taking a statistics test or attending a statistics class); (d) computational self-concept (anxiety related to mathematical calculations); (e) fear of asking for help (anxiety experienced when help seeking is considered); and (f) fear of statistics instructors (negative attitudes toward statistics teachers). A similar typology of statistics anxiety was also proposed by Onwuegbuzie (1997).

There is overwhelming evidence that motivation influences learners’ statistics anxiety, learning experience, and performance in statistics courses (Lavasani et al., 2014; Onwuegbuzie & Seaman, 1995; Onwuegbuzie & Wilson, 2003). For example, when learners are anxious about underperforming in an exam, this negative emotion can impact their ability to employ appropriate metacognitive strategies. On the other hand, when learners are intrinsically motivated to learn the content, they are more likely to develop positive emotions and employ appropriate learning strategies (Linnenbrink-Garcia et al., 2016). Despite the interactions between motivation and anxiety, there is a lack of research on how theories of motivation can be used to develop learning analytics tools, such as learning analytics dashboards, to support learners’ motivation (Bodily & Verbert, 2017; Lonn et al., 2015) and learning performance in anxiety-inducive courses such as statistics. This practical and theoretical gap guided the design and implementation of the present study.

Motivation

Achievement goal theory

Achievement goal theory was used in the conceptualization of this study because it recognizes the influence of learners’ goal orientations on cognitive and affective outcomes (Ames, 1992; Elliott & Dweck, 1988). From a traditional perspective of achievement goal theory, learners can adopt two types of goal orientations: performance or mastery goals. A mastery goal orientation focuses on individual progress and learning, while a performance goal orientation focuses on relative ability and comparison with other learners (Ames, 1992; Pintrich, 2000b).

Elliot and colleagues expanded the dichotomic conceptualization of mastery versus performance goal orientations by including two valences (i.e., approach and avoidance) to the traditional mastery and performance goal orientations (Elliot & Harackiewicz, 1996; Elliot & McGregor, 1999; Elliott & Dweck, 1988). Based on this newer 2 × 2 model of goal orientation, learners with performance-approach goals want to demonstrate ability and competency in comparison to others – normative goal theory (Harackiewicz et al., 2002) whereas learners with performance-avoidance goals do not want to be perceived as incompetent. Similarly, learners with mastery-approach goals focus on the mastery of the content while those with mastery-avoidance goals worry about not learning the content (Pintrich, 2000b). In addition to this 2 × 2 goal orientation model, another model involves self-based approach goals (e.g., Personal-Best goals, 3 × 2 model: task goals, self-goals and other goals), which focus on learners’ prior performance and experiences (Elliot et al., 2011). It is worth noting that self-based goals align with mastery goals, in which performance is based on the individual’s own learning progress in relation to prior performance.

Finally, more contemporary conceptualizations of goal orientation combine mastery and performance goals within learners (Barron & Harackiewicz, 2001; Darnon et al., 2010). For example, students may have an intrinsic interest in statistics (i.e., mastery goal orientation) and still pay close attention to their grades on quizzes and exams (i.e., performance goal orientation) as a way of monitoring their learning throughout the course and adjusting their metacognitive strategies as needed. This approach is referred to as the multiple goal perspective in contrast to the traditional normative goal theory, which proposes that mastery and performance goal orientations are mutually exclusive (Harackiewicz et al., 2002). The learning analytics dashboards used in this study subscribe to the multiple goal perspective, which combines performance and mastery goal orientation elements.

Learning analytics dashboards

Advances in technology to record and store unprecedented amounts of learners’ data enable enhanced evaluation of pedagogical practices through the use of learning analytics approaches. Learning analytics involves the “measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” (Siemens & Gasevic, 2012, p. 1). As an application of learning analytics, dashboards are helpful data visualization scaffolds that are being increasingly adopted in online and post-secondary education contexts (Schwendimann et al., 2017; Siemens & Gasevic, 2012).

Dashboards help to convey complex quantitative information to the user (instructor, student, course designer, etc.) in a visual form and with digestible, actionable items. For example, instead of providing numeric data on individual assignments, many dashboards summarize these data visually to emphasize overall trends for each module. It is important, however, that researchers and instructors recognize that they bring their own perspectives, epistemological stands, as well as biases as they design, develop, and evaluate dashboards (Knight & Shum, 2017).

It is worth noting that there is a lack of standardization regarding the definitions and terminologies used to describe learning analytics dashboards, which reflects the interdisciplinary and incipient nature of the field (Bodily & Verbert, 2017; Schwendimann et al., 2017). Some of the most common terms used for these applications include ‘learning dashboards’, ‘educational dashboard’, ‘dashboard for learning analytics’, ‘data dashboard’, ‘web dashboard’, and ‘learning analytics dashboards’(Schwendimann et al., 2017).

Despite the increasing use of learning analytics dashboards, the body of literature around these applications is relatively small; the field is seen as an incipient area of research under the umbrella of learning analytics (Bodily & Verbert, 2017). The need for more evidence-based research is critical as educators design, implement, and test learning analytics dashboards and their influence on students’ motivation, interest, self-regulation, awareness, self-efficacy, and learning outcomes (Bodily & Verbert, 2017; Jivet et al., 2018; Valle et al., in press). However, most dashboard studies are not framed using theories of motivation, self-efficacy, regulation, and other relevant concepts and so their contributions to the development of theories of how dashboards should be designed and used are limited (Jivet et al., 2018).

Conceptual framework

To investigate the potential of dashboard applications as technology-enhanced pedagogical tools to support learners’ cognitive and affective outcomes (Zimmerman, 2008), we adopted a conceptual framework that highlights a multiple goal orientation perspective, where mastery and performance goal orientations elements were used in the design of our descriptive and predictive learning analytics dashboards (Harackiewicz et al., 2002).

Mastery goal orientation is reflected in the emphasis on the content to be learned, specifically the names of topics shown in the graph’s x-axes, which are used in both the predictive and descriptive dashboards. Performance goal orientation is reflected in the design of the performance feedback presented to learners, which is either norm-referenced in the descriptive dashboard or self-referenced in the predictive dashboard. Norm-referenced feedback compares an individual student’s performance to the performance of the class, whereas self-referenced feedback compares a student performance on a task relative to this same student’s prior performance (Biesinger & Crippen, 2010; Koenka & Anderman, 2019).

Our study’s conceptual framework (Fig. 1) reflects the current gaps in the literature regarding the cognitive and affective influence of descriptive vs. predictive dashboards on learners (Corrin & de Barba, 2014; Gašević et al., 2015) and informs our research questions:

Fig. 1

Graphical depiction of the conceptual framework employed by this study

To what extent does the use of a predictive or descriptive learning analytics dashboard in an online statistics course influence learners’:

RQ1 – motivation?

RQ2 – statistics anxiety?

RQ3 – performance outcomes?

RQ4 – How do learners perceive their learning experience, motivation, and statistics anxiety while using predictive or descriptive learning analytics dashboards?

Materials and methods

Research design

This study used quantitative and qualitative data (Creswell et al., 2003; Johnson & Onwuegbuzie, 2004) to provide a more comprehensive account of the use of predictive and descriptive dashboards and their influence on learners’ statistics anxiety, motivation, and learning performance outcomes in an online statistics course. The study was implemented in two semesters with two different cohorts of participants who were randomly assigned to three groups (control, descriptive, and predictive) and the outcomes for each group were compared. Following the IRB approved protocol, we collected the following quantitative information: number of dashboard views, prior content knowledge scores, prior experience with programming languages, quiz results, final grades, and pre and post results (beginning and end of the semester, respectively) from the Motivation Strategies for Learning Questionnaire scale – MSLQ (Pintrich et al., 1991, 1993) and from the Statistical Anxiety Rating Scale – STARS (Cruise et al., 1985). Qualitative data included open-ended surveys questions at the end of the semester and 10 semi-structured individual interviews (Seidman, 2006). While semi-structured individual interviews were used to collect qualitative data only in one semester, open-ended survey items were used in both semesters.

Context

The study was implemented in an introductory online statistics graduate course at a large public university in the southeastern United States during Fall 2018 and Spring 2019. The course uses Canvas™ as the learning management system and is organized into 14 modules covering different statistical concepts. The course is offered twice a year and features video lectures, tutorials, required and optional readings, discussion forums, weekly data and conceptual quizzes, and a final exam. Two instructors, co-authors in this study, facilitate the course and online learning interactions.

Participants

A convenience sample of 179, representing 69% of all the students (179/261) who were invited to participate in the study, completed the study (control = 69, descriptive = 56, and predictive = 54). The majority of participants were female (59.22%,) and Caucasian (55.87%), with a mean age of 28.16 (standard deviation = 5.80). The majority of students were pursuing a master’s degree (63.13%), and a large set of the participants had no programming experience (55.87%). When asked about reasons for taking the course, most participants (74.86%) selected the option “Will help improve my academic skills”. A 12-question pre-test assessed students’ prior knowledge of statistical concepts such as t-tests and p-values. Participant mean performance on the pre-test, based on the proportion of correct responses, did not suggest any systematic differences between groups: control = 64.73%, descriptive = 65.77%, and predictive = 65.74%.

Design and development

The predictive and descriptive learning analytics dashboards used as the intervention in this study were developed using R software (version R 3.5.2 – ©2004–2016) and Shiny™, a web development framework that enables the visual display of results from R. The design of the dashboards was based on the multiple goal orientation perspective (Harackiewicz et al., 2002), where elements from the performance approach (e.g., focus on predicted letter grades) and the mastery goal approach (e.g., focus on the content or skills to be learned) orientations are combined to support learners’ motivation. Importantly, the use of visual feedback based on learners’ performance in the course was included in the design of the dashboards because it supports learners’ formative assessment and self-regulated learning (Fabiano et al., 2018; Nicol & Macfarlane‐Dick, 2006). Thus, the design of the dashboards were based on the premise that monitoring of performance is related to metacognitive processes, self-regulated learning, and motivation profiles (Baars & Wijnia, 2018). For example, learners may use the information about their performance to regulate how much time and effort they should spend studying the course material.

Predictive dashboard design and development

In the predictive dashboard, performance goal orientation elements were represented by the predicted probability of learners’ receiving letter grades A, B, C, or D/E at the end of course (Fig. 2). The letter grades were color-coded to facilitate their identification in the dashboard. Letter grade predictions were updated weekly as the semester progressed according to learner performance on weekly quizzes, using a Naïve Bayes model, a popular machine learning method presented in Appendix A. The mastery goal elements were represented by the focus on the content to be covered in each module (e.g., summary statistics, ANOVA) and on the “learning progression” of individual learners (Fig. 2). For example, the green line in Fig. 2. shows that, based on the quizzes results up to module 5, the probability of this particular individual receiving a letter grade A at the end of the course is 78% whereas the probability a B grade is 21%. This dashboard represented a self-referenced design approach, where the predictions were pertinent to individual learners only, without class comparison elements such as predictions about other students in the class. Based on the goal orientation literature, a self-referenced dashboard can reduce competition and anxiety among learners as the focus is on the individual learner and his or her own progress in the course (Elliot & McGregor, 1999; Lavasani et al., 2014). To inform students about the inherent limitations of predictive models, the following disclaimer was included in the dashboard display “These probabilities are based on data from previous semesters of students with similar performance and, therefore, are not 100% accurate”.

Fig. 2

Predictive learning analytics dashboard

Descriptive dashboard design and development

The descriptive dashboard was conceptualized to represent a more traditional learning analytics dashboard design. More specifically, descriptive data were used to display students’ learning performance per module relative to the average performance of the class. For example, the solid black line in Fig. 3. shows that a student received 67% of points in module 5, while the class average was about 90% for the same module. Average grade and grades distribution are often used as result-related indicators in learning analytics dashboards (Schwendimann et al., 2017). Similar to the predictive dashboard, the descriptive dashboard also displays mastery goal orientation elements (e.g., content covered in the modules, shown in the x-axis) and the representation of “learning progress” conveyed by the solid black line.

Fig. 3

Descriptive learning analytics dashboard

Control group

Learners in the control group only had access to the standard Canvas™ dashboard, which presented descriptive data (e.g., mean scores) and a boxplot with the student’s score per assignment (Fig. 4).

Fig. 4

Default dashboard available through Canvas™

All dashboards were updated weekly based on weekly quizzes and participants from all groups, including the control group, received weekly reminders about these updates. Participants in the Descriptive and Predictive groups could access their dashboard whenever they wanted through a link to the web application.

Data sources and instruments

Quantitative measures

In addition to total quiz scores and final exam scores, we collected pre and post results from the Motivation Strategies for Learning Questionnaire motivation scale – MSLQ (Pintrich et al., 1991, 1993) and from the Statistical Anxiety Rating Scale – STARS (Cruise et al., 1985; Hanna et al., 2008).

The MSLQ instrument was selected because it is based on the social-cognitive model of motivation, which is the framework underlying the achievement goal theory (Pintrich, 2000a, 2000b). Furthermore, MSLQ addresses affective variables that are relevant to the present study, including mastery and performance goal orientations as illustrated by the following items: “The most satisfying thing for me in this course is trying to understand the content as thoroughly as possible” and “Getting a good grade in this class is the most satisfying thing for me right now”, respectively. The MSLQ instrument has been used in many empirical studies, which offers further evidence of the scales’ construct validity (Schutz et al., 1998; Vahedi et al., 2012).

The MSLQ instrument has two scales: a cognitive strategies scale, used to measure metacognitive learning strategies; and a motivational scale, used to measure motivational orientations (Pintrich et al., 1993). For this study, we used only the motivational scale, which has 31 items representing six factors and 3 areas of the motivation construct: (a) value, (b) expectancy, and (c) affective components. The value component subscale measures learners’ goals and value beliefs and consists of three subconstructs: (a) intrinsic goal orientation, (b) extrinsic goal orientation, and (c) task value. The expectancy component subscale reflects learners’ expectancy beliefs for completing the task and corresponds to two subconstructs: self-efficacy and control beliefs for learning. Finally, the affective component subscale addresses test anxiety (Pintrich et al., 1993) and does not have any subconstruct. The instrument was presented using a 7-point Likert-style scale, where 1 and 7 represented “Not at all true of me” and “Very true of me,” respectively. For each participant, final scores for each factor were computed by summing the items representing that factor and taking an average. Thus, final scores for each factor range from 1 to 7. Cronbach’s alpha results for the MSLQ instrument ranged from 0.65 to 0.94 for the pre-surveys and from 0.67 to 0.95 for post-surveys. There were only small differences regarding Cronbach’s alpha values from pre- to post-surveys within subconstructs. All Cronbach’s alpha values were above 0.70, except for the external motivation subconstruct (pre = 0.65; post = 0.67) as shown in Appendix B.

The STARS instrument has been widely used to measure statistics anxiety (Hanna et al., 2008; Onwuegbuzie & Wilson, 2003). Furthermore, recent confirmatory factor analysis studies have discussed the importance of testing STARS accross different contexts (e.g., face–to–face vs. online) and cultures, and have provided further evidence of the instrument’s construct validity and internal consistency reliability supporting the original six-factor structure of the instrument (DeVaney, 2016; Papousek et al., 2012). More specifically, the instrument has 51 items, representing 6 factors: (a) worth of statistics, (b) interpretation anxiety, (c) test and class anxiety, (d) computational self-concept, (e) fear of asking for help, and (f) fear of statistics teachers. We used the STARS version adapted by Hanna et al. (2008) and the instrument was presented using a 5-point Likert-style scale. The first 23 items focused on situations that may trigger anxiety on learners and were presented on a scale from 1 (No anxiety) to 5 (Great deal of anxiety). The remaining 28 items focused on learners’ perceptions of statistics and statistics instructors and were also presented on a scale from 1 (Strongly disagree) to 5 (Strongly agree). Final scores for each factor were computed by summing the items within each factor and taking an average. Cronbach’s alpha results for the STARS instrument ranged from 0.77 to 0.93 for the pre-survey and from 0.89 to 0.94 for the post-survey (Appendix B).

Qualitative measures

The qualitative data sources included open-ended survey questions and 10 individual semi-structured interviews, which were used to understand the meanings participants made from their experience related to using the dashboards in the course (Larkin & Thompson, 2012).

The selection of participants for the interviews was based on a convenience sample from those students who had already completed the pre-survey on their motivation and statistics anxiety. We used a purposeful sampling strategy to ensure representation of all groups (control, predictive or descriptive dashboards) and different genders (Palinkas et al., 2015). Seven females and three males agreed to participate in the interviews. The interviews lasted between 12 min and 33 min (mean = 18.5 min) and were conducted by two of the authors. Although most of the interviews were face-to-face, there were a few online interviews (n = 3). It is worth noting that each interviewer conducted interviews with at least one participant from each group. This was a deliberate decision to account for the role of the researcher as an instrument (Creswell & Poth, 2018). Following the IRB approved protocol, pseudonyms were used in the transcripts to protect the privacy of the participants. The interview protocol included questions such as: “How often do you use [x dashboard] in this course? What has been your experience using it in this course? Did you find any advantage or disadvantage in using this resource?

Data analysis

Quantitative data analysis

The data were organized and analyzed using R programming language (Version R 3.5.2). Only participants who completed pre- and post-surveys and used the dashboards at least once (descriptive and predictive groups) were included in the analysis. Using a Wilcoxon rank sum test, we found no statistically significant differences (W = 1409.50, p = 0.538) between the predictive (Mdn = 3.50) and descriptive groups (Mdn = 3.00) in relation to the number of times learners checked the dashboards. The number of times the dashboard was used by individual learners was not included in our models because that would increase the number of parameters being estimated, making the estimation of the individual parameters and interpretation of our model results more challenging.

Model for learning performance. To evaluate if learning performance was influenced by the treatment, we relied on a Tobit regression model. We relied on this model because a student obtaining the maximum score of 100 only tells us that the true score/performance of the student was at least 100 but could have been higher if the measurement instrument was not limited to a 0–100 scale. By acknowledging that both total quiz and final exam scores are bounded between 0 and 100, this model avoids the bias in parameter estimates that can be is introduced due to ceiling effects when using a regular ANOVA/regression model (Wang et al., 2008).

Let and be the observed score and the partially latent score, respectively, of learner i. The Tobit regression model assumes that if . However, this model assumes that (i.e., true performance is higher than 100) if and that (i.e., true performance is lower than 0) if , essentially treating the observed scores as censored observations whenever they are equal to 0 or 100. The Tobit model further assumes that arises from the following regression (1):

where and are binary variables indicating if learner i was assigned to the descriptive or predictive treatment. In this expression, and are the regression intercept and slope parameters, respectively.

Model for mslq and stars. We adopted a similar Tobit model to analyze the results from the MSLQ and STARS constructs, given that these scores are also bounded between 1 and 5 (MSLQ) and 1 and 7 (STARS) and that some students have obtained the minimum and maximum possible scores. This Tobit model acknowledges that a student that scored 7 for MSLQ might have a true motivation that is actually higher than 7 but that, due to the range of the instrument, was constrained to be equal to 7. Similarly, a student that scored 5 for STARS might have a true anxiety that is actually higher than 5. In statistics, these outcomes are called censored observations because they provide only partial information on the phenomenon of interest. Furthermore, we modified the standard Tobit regression model to accommodate for the fact that both the pre-intervention scores and post-intervention scores are potentially censored.

Let and denote the partially latent pre and post-scores, respectively, for learner i. We assume that the observed post-score is given by (2):

where F and C stand for the minimum (floor) and maximum (ceiling) value that the construct can take. Similarly, we assume that the observed pre-score follows the same rules (3):

We rely on the following regression to model as seen in (4):

Because and are partially latent, we have to estimate these variables whenever the observed scores are at the ceiling (i.e., or ) or at the floor (i.e., or ).

Priors and model fitting. We fit these models in a Bayesian framework. For the Tobit model applied to learner’s performance, we specified the following conditionally conjugate priors (5):

The priors for and are relatively standard uninformative priors but we adopt a more informative prior for and . This informative prior tends to shrink to zero, reducing the chance of finding spurious significant results.

For the modified Tobit model applied to the MSQL and STARS constructs, we adopted the following priors (6):

Again, the priors for are relatively standard uninformative priors but we relied on more informative priors for the parameters associated with the treatments (i.e., ) to reduce the chance of obtaining spurious significant results. Finally, for this model, we also needed to specify the prior for . We assumed that (7):

These models were fit with a Gibbs sampler algorithm, using customized R code. Furthermore, we calculate a measure of evidence, similar to p-values, the minimum of the posterior probability that is smaller or greater than zero (i.e., . Finally, we performed a sensitivity analysis where the informative priors were replaced by uninformative priors and the models were re-fit to the data. This analysis revealed that our statistically significant findings were robust to the choice of prior.

Qualitative data analysis

The search for “repeated patterns of meaning” (Braun & Clarke, 2006, p. 15) in the thematic analysis (Braun & Clarke, 2006) involved transcribing the interviews, coding the data, and theme generation. The interviews were transcribed verbatim and initial codes were generated. These initial codes were then compared and contrasted with the researchers’ reflective notes and refined after discussion with peer debriefers to minimize research bias and increase trustworthiness of the analysis. Finally, similar codes were combined into themes and discussed for further validation. The process of engaging participants in the interviews was facilitated by the common shared experiences we had with taking statistics courses in the past. Nevertheless, we attempted to bracket our experiences via personal reflection journals and feedback during the peer debriefing process.

Results

We present the results and provide a discussion of these results based on the four research questions addressed in this study.

RQ1. To what extent does the use of a predictive or descriptive learning analytics dashboard in an online statistics course influence learners’ motivation?

The Bayesian multiple regression model for the final intrinsic goal orientation showed a positive statistically significant effect of the interaction of the predictive dashboard and the initial level of intrinsic goal orientation (. This interaction revealed that the effect of the predictive dashboard changed according to the students’ initial intrinsic goal orientation scores (learners’ motivation to learn the content). More specifically, learners in the predictive dashboard who started with higher intrinsic goal orientation scores (e.g., pre-intrinsic goal score = 6) had, on average, 0.10 higher intrinsic goal orientation scores at the end of the semester when compared to learners in the control group. On the other hand, learners in the predictive dashboard who started the course with lower intrinsic goal orientation scores (e.g., pre-intrinsic goal score = 2) had on average 0.86 lower intrinsic goal orientation scores at the end of the semester when compared to learners in the control group (Fig. 5b).

Fig. 5

Relationship between pre and post “Intrinsic Goal Orientation” scores for students in the control, descriptive, and predictive treatments. Panel (a) compares the results for the control (grey) and the descriptive treatment (red) while panel (b) compares the results for the control (grey) and the predictive treatment (red). There was no statistically significant effect of the descriptive dashboard on intrinsic goal orientation (panel a). However, there was a statistically significant interaction indicating that the effect of the predictive dashboard on intrinsic goal orientation depended on the initial intrinsic goal orientation of students (panel b). Lines are posterior mean while envelopes represent 95% pointwise credible intervals (Color figure online)

There were no statistically significant differences for the other motivation outcomes, namely extrinsic goal orientation, task value, control beliefs, self-efficacy, and test anxiety (Table 1).

Table 1

Results from the Bayesian modified Tobit models for motivation constructs

MSLQ construct	Parameter	β	p
Intrinsic goal orientation	Intercept	2.09	<.001***
	Descriptive	−0.32	.320
	Predictive	−1.34	.023*
	Pre-score	0.58	<.001***
	Pre-score x Descriptive	0.09	.247
	Pre-score x Predictive	0.24	.026*

*p < .05. ***p < .001. Results related to other motivation constructs that were not statistically significant can be found in Appendix C

RQ2. To what extent does the use of a predictive or descriptive learning analytics dashboard in an online statistics course influence learners’ statistics anxiety?

The Bayesian multiple regression model revealed a negative statistically significant effect of the predictive dashboard on final interpretation anxiety scores ( −). Interpretation anxiety is a construct that measures learners’ anxiety to deal with data or solve statistical problems. This result suggests that, compared to learners in the control group, the predictive dashboard decreased learners’ interpretation anxiety. More specifically, the model indicated that learners in the predictive dashboard had on average significantly lower interpretation anxiety scores at the end of the course than those in the control group. For example, learners with pre-interpretation anxiety scores of 2 are predicted to have, on average, 0.12 lower post-interpretation anxiety scores in the predictive group compared to the control group. It is worth noting that the effect of the interaction of the predictive dashboard group and pre-interpretation anxiety scores approached statistical significance ( −). This interaction helps to explain why there was also a noticeable increase in final interpretation anxiety scores for the predictive group for those who started the course with higher levels of interpretation anxiety in comparison to the control group. For example, learners with pre-interpretation anxiety scores of 4 are predicted to have, on average, 0.28 higher post-interpretation anxiety scores in the predictive group compared to the control group (Fig. 6b).

Fig. 6

Relationship between pre and post “Interpretation anxiety” scores for students in the control, descriptive, and predictive treatments. Panel (a) compares the results for the control (grey) and the descriptive treatment (red) while panel (b) compares the results for the control (grey) and the predictive treatment (red). There were no statistically significant differences between the descriptive and control groups (panel a). On the other hand, the predictive dashboard tended to reduce interpretation anxiety in comparison to the control group (panel b) but this effect depended on the pre-interpretation anxiety scores. Lines are posterior mean while envelopes represent 95% pointwise credible intervals (Color figure online)

The model with final worth of statistics anxiety, a construct that measures negative attitudes toward statistics and perceived usefulness of statistics, revealed a statistically significant interaction effect (dasboard x initial worth of statistics anxiety scores) for both, the descriptive ( −) and predictive ( −) dashboards, indicating that final worth of statistics scores depended on learners’ initial levels of this anxiety. This effect is particularly evident for those learners who started the course with higher levels of worth of statistics anxiety (e.g., pre-worth anxiety = 4). For these learners, being on the predictive or descriptive dashboards was helpful to lower their negative attitudes towards statistics (worth of statistics) by an average of 0.75 at the end of the course compared to learners in the control group (Fig. 7).

Fig. 7

Relationship between pre and post “Worth of Statistics anxiety” scores for students in the control, descriptive, and predictive treatments. Panel (a) compares the results for the control (grey) and the descriptive treatment (red) while panel (b) compares the results for the control (grey) and the predictive treatment (red). Both of the descriptive (panel a) and predictive (panel b) dashboards reduced worth of statistics anxiety in comparison to the control group for learners who started the course with higher levels of worth of statistics anxiety

Although we found statistically significant differences between groups for interpretation and worth of statistics anxiety, there were no statistically significant differences for the other anxiety constructs, namely test and class anxiety, fear of asking for help, computational self-concept, and teacher anxiety (Table 2).

Table 2

Results from the Bayesian modified Tobit models for statistics anxiety constructs

STARS construct	Parameter	β	p
Interpretation anxiety	Intercept	0.79	<.001 ***
	Descriptive	−0.06	.408
	Predictive	−0.52	.041*
	Pre-score	0.62	<.001 ***
	Pre-score x Descriptive	−0.04	.378
	Pre-score x Predictive	0.20	.053
Worth of statistics	Intercept	0.12	.223
	Descriptive	0.29	.085
	Predictive	0.29	.096
	Pre-score	0.96	<.001 ***
	Pre-score x Descriptive	−0.26	.040*
	Pre-score x Predictive	−0.26	.046*

*p < .05. ***p < .001. Results related to other statistics anxiety constructs that were not statistically significant can be found in Appendix D

RQ3. To what extent does the use of a predictive or descriptive learning analytics dashboard in an online statistics course influence learners’ performance?

The Tobit model used to compare the effect of dashboard type (predictive, descriptive, and control) on total quiz scores and final exam scores indicated no evidence for an effect of the learning analytics dashboards on learning outcomes for neither final exam ( nor the total quiz scores ( as shown in Table 3. The descriptive statistics related to learning performance outcomes per group are shown in Table 4. Although not statistically significant, learners in the descriptive and predictive groups generally did better than those in the control group, especially in relation to final exam scores.

Table 3

Results from the Bayesian modified Tobit models for learning performance

Learning performance	Parameter	β	p
Final exam	Intercept	7.97	<.001***
	Descriptive	0.52	.057
	Predictive	0.14	.324
Total quiz	Intercept	9.19	<.001***
	Descriptive	0.11	.157
	Predictive	0.03	.415

***p < .001

Table 4

Means (M) and standard deviation (SD) of final exam scores and total quiz scores by group

Learning performance	Control M (SD)	Descriptive M (SD)	Predictive M (SD)
Final exam	78.51 (20.56)	83.82 (17.57)	80.94 (14.21)
Total quiz	91.88 (6.55)	93.01 (5.08)	92.21 (6.10)

RQ4. How do learners perceive their learning experience, motivation, and statistics anxiety while using the predictive or descriptive learning analytics dashboards?

The themes that emerged as part of the qualitative data analysis are described below.

Focusing on what I’m learning

Although we did not find robust results regarding differences in learning outcomes between groups (control, descriptive, and predictive dashboards), most participants reflected on their performance as they described the course and its resources. Mastery approach to learning was also implied by some participants. For example, “Tom”, who used the descriptive dashboard, was very enthusiastic about learning statistics and defined statistics as “a very useful tool […] it’s useful not only in science, it’s also useful in business.” After discussing the utility value of statistics, he concludes his rationale for taking the course by saying “Well, you have to learn statistics well.” Tom did not seem to be negatively affected by the norm-referenced design of the descriptive dashboard as he mostly focused on his individual scores, stating “[…] class average doesn’t really bother me, right. I’m more concerned about my own score”.

“Patricia”, on the other hand, reflected on her mastery of the content when she talked about the possible advantages of using the descriptive dashboard, describing it as a more comprehensive way to provide information to students:

For me, like I said, it really helps put in perspective what I’m not understanding. It really puts into perspective, like, looking at the numbers on the Canvas grade is one thing but looking at it on the [descriptive] graph, this helps me track: Am I starting to kind of fall on the modules? Am I understanding them or not? This really puts in perspective how well I’m doing, what my areas of weaknesses are, and I think it really puts that in perspective.

The perception of the dashboards as helpful metacognitive tools to regulate learning (e.g., monitoring, motivation) was echoed by learners’ responses to the open-ended items on the post survey that was given at the end of the semester. For example, a male student who used the predictive dashboard said “It was great. I enjoyed watching the week-to-week progress.”

Performance matters to me

All of the participants reflected on their experiences from a performance approach perspective. This was true even for those who prioritized the learning of the content (mastery approach). It is important to note that the word “grade” was purposefully avoided during the interviews to give participants an opportunity to frame “progress in the course” based on their own experiences and perceptions. For example, “Jim”, who used the predictive dashboard, mentioned that if his scores were not good, he would probably check the dashboard more often:

Yeah, I kinda went away after that [I realized] I’m doing pretty good in the course, so it wasn’t necessarily changing my studying behavior. I would say perhaps if I was doing bad, [it would work] more like an eye-opener and [I would] say oh well, yeah, I really need to study, but since I’m doing okay, and I probably don’t see myself changing how I study per say, it was more like an interesting [artifact], but that was kind of the extent of it.

“Jen”, a learner who used the descriptive dashboard throughout the semester, described that she returned to view the graph “because I want to be doing as well as everybody else, if not better.” The performance-approach goal orientation can be observed through Jen striving to do well in comparison to the other learners in the course.

“Carol” was concerned about her performance in the course as well. She mentioned that she made some mistakes “because of vocabulary,” as English is not her first language. Reflecting on her experience in the course, she said that she had some mixed feelings about the predictive dashboard as it contributed to her frustration. Carol’s anxiety was based on her fear of a bad performance (performance-avoidance goal orientation). This example corroborates the literature regarding the relationship between performance orientation and anxiety (Pintrich, 2000b), which can be observed in the comment in which Carol, when prompted about her feelings while checking the predictive dashboard, said “I thought I would be much worse. It was kind of [a] relief but also it made me anxious”.

Based on the responses to open-ended items from the post-survey, these feelings of anxiety and frustration were also experienced by other learners. For example, a male student who used the descriptive dashboard said the following:

Let me tell ya, finding out that I was doing below average did a number on my morale and didn’t really motivate me to work harder so much as to confirm my preexisting belief that I just don’t have the capacity to do math or stats. But hey I got to help make everyone above the curve feel better about themselves so that’s a silver lining right there at least.

The norm-referenced features of the descriptive dashboard were emphasized by many other students. For example, a female student said “I found it useful because I was able to know how my grades compared to grades of other students. I wish it could be updated more rapidly.”

It “holds a controversial place in my mind”

A female student emphasized her mixed feelings about the predictive dashboard by saying “It gave me great anxiety at first, but then I found that it pushed me to try harder!” These mixed feelings were echoed by another participant, who said the following about her experience with the descriptive dashboard:

Although I enjoyed the “How am I doing” graph, I would get a sense of failure or embarrassment if my weeks points dipped below the class average. Conversely, I would also feel a sense of pride when I was above the class. It was a bit of a double-edged sword.

Another comment by a female student with an apparent combination of performance approach and performance avoidance goal orientations emphasized the norm-referenced design of the descriptive dashboard as an element that induces anxiety:

The “How am I doing?” graph holds a controversial place in my mind. I am not a competitive person, but I am accustomed to performing above average and, thus, seeing this brings me a degree of comfort. However, when I do below average it is the worst thing to see! The distress that brings me is difficult to describe. I can see how it would be a useful motivator to some, but in the harder modules I just stopped looking at it because I didn’t want to risk seeing that I was comparatively performing poorly.

I didn’t get it at first

Most students who participated in the interviews did not understand the predictive and descriptive dashboards when they first explored them. For example, Patricia, who used the descriptive dashboard, said the following:

It was a little bit confusing for me at first just because I wasn’t really sure what I was looking at first, and then I kind of had to read what these were and, but once I actually read everything and I was able to understand it pretty easily because these are the modules, right?

“Emma”, who used the predictive dashboard, was also confused at first. Like Patricia, she was able to understand the graph after reading the directions “So I was kind of confused by it at first, but then obviously like reading the directions helped a lot […] It just took me a while to piece out what everything meant”.

While Patricia and Emma were initially confused about the graphs, other students, like Jim, had a different experience. He described his experience with the predictive dashboard by saying “Yeah. I think it was pretty clear. I didn’t think I needed, like, a whole lot of additional explanation to really understand what was trying to be conveyed.”

Discussion

In this article, we describe an experimental study conducted over two semesters based on the multiple goal orientation theory to illuminate how and whether mastery and performance goal orientation approaches embedded in learner-centered learning analytics dashboards can influence learners’ motivation, anxiety, and learning performance. The adoption of the multiple goal orientation perspective (Harackiewicz et al., 2002) enabled us to use these concepts in the design of the (self-referenced) predictive and (norm-referenced) descriptive dashboards. The use a conceptual framework also gave us the necessary theoretical frame to interpret the influence of these design elements on learners’ cognitive and affective responses (Antonenko, 2015; Jivet et al., 2018). The attention to, and evaluation of, learners’ affective outcomes as an integral part of the study represents an important practical and theoretical contribution because affective outcomes have been neglected in most learning analytics dashboard studies despite their influence on learners’ satisfaction and learning outcomes, particularly in computer-based learning environments (Duffy & Azevedo, 2015; Jivet et al., 2018; Schunk et al., 2014; Zimmerman, 2008).

Motivation

Results from this study show that the predictive learning analytics dashboard was beneficial to learners who started the course with higher intrinsic motivation; however, it reduced motivation for those who started the course with lower motivation. While the positive influence of the predictive dashboard to support motivation among highly motivated learners did not come as a surprise given its self-referenced approach, the fact that the predictive dashboard had a detrimental effect for learners with lower motivation was intriguing. This is interesting because the self-referenced approach emphasizes learning progress relative to the individual’s own progress in the course, which has been described as an optimal pedagogical approach to support intrinsic motivation (Elliot & McGregor, 1999; Lavasani et al., 2014).

The detrimental effects of the predictive dashboard for learners with lower motivation may be due to learners’ focus on performance goal orientation elements and a possible demoralizing effect created by the dashboard. Additionally, learners’ lack of familiarity with predictive models and how they function may also have impacted learners’ motivation. In particular, some students may have expected predictions that were more aligned with their grades in a given module when, in reality, some modules and assignments are more influential than others in the predictive model. For example, earning 100% on a relatively easy module in which everybody usually does very well is unlikely to substantially increase the predicted letter grade whereas relatively small performance differences in a more challenging module may dramatically influence predicted outcomes. This lack of familiarity with how predictive models work was reflected in the feedback by a female student who said “It [predictive dashboard] was nice, but at times made me nervous. Even when my grade was overall very high, it still showed a high possibility of me getting a B in the class, which was a little nerve wracking.”

A question that remains, however, is why would the predictive dashboard be more detrimental for some learners than others? It is possible that the lack of contextual, norm-referenced information may have contributed to an increase of self-oriented perfectionism among less motivated learners. Self-perfectionism refers to the “intrapersonal dimension of perfectionism,” where learners may develop unreasonable high standards for their performance, which may lead to self-criticism and decreased intrinsic motivation (Mills & Blankstein, 2000; Miquelon et al., 2005). One way to alleviate the focus on the self-criticism would be to provide additional scaffolding to help less motivated learners to better contextualize their academic performance (e.g., norm-referenced feedback). The provision of scaffolds based on learners motivational profiles has been addressed in previous studies involving Computer-Supported Collaborative Learning (Rienties et al., 2012) and could be used in studies with learning analytics dashboards to further explore the effects of intrapersonal motivational factors (Lavasani et al., 2014; Rosenzweig & Wigfield, 2016).

The predictive dashboard reduced interpretation anxiety among learners in comparison to the control group. Interpretation anxiety refers to the anxiety learners feel as they need to interpret statistical analysis or make decisions based on statistical data (Cruise et al., 1985). This result is particularly interesting because it corroborates other studies that describe how self-referenced approaches, such as the one used in the predictive dashboard, are more likely to reduce competition and anxiety among learners (Elliot & McGregor, 1999; Lavasani et al., 2014). However, it is worth noting that these studies also emphasized that norm-referenced approaches, such as the one used in the descriptive dashboard, are more likely to induce anxiety among learners (Elliot & McGregor, 1999; Lavasani et al., 2014), which was not the case based on the anxiety measures used in this study. In fact, both the descriptive and predictive dashboard reduced worth of statistics anxiety in comparison to the control group, indicating that dashboard designs that combine mastery goal and performance goal orientation elements can be employed to reduce learners’ negative attitudes towards statistics, particularly among learners who start the course with higher levels of worth of statistics anxiety.

These findings have important implications for theory and practice. From a theoretical perspective, these findings support multiple goal theory by providing evidence that students can indeed benefit (through a decrease in interpretation anxiety and worth of statistics anxiety) from the co-existence of performance and mastery goal orientation elements (Darnon et al., 2010; Dela Rosa & Bernardo, 2013; Harackiewicz et al., 2002). Similar to the findings described by Darnon et al. (2010), our results offer additional evidence that social comparison orientation, which refers to the tendency to seek social comparison information, can play an important role on how people regulate their learning experiences (Schunk & Greene, 2018), including their goals for mastering statistical content. From a practical perspective, the descriptive and predictive learning analytics dashboard used this study can serve as tested model for future learning analytics dashboard developments intended to reduce statistics anxiety.

Learning outcomes

Despite the combination of mastery and performance goal orientation elements in the design of the predictive and descriptive dashboards and the influence they had on learners’ intrinsic goal orientation, interpretation anxiety, and worth of statistics anxiety, we did not find statistically significant differences between groups in relation to final exam or total quiz scores.

The lack of statistical significance for learning performance outcomes in this study aligns with the results found by Biesinger and Crippen (2010), who used a similar graph design approach based on self-reference and norm-referenced elements as distinct feedback protocols in the context of a chemistry course. In their discussion of the results, they acknowledged that one possible reason for the non-significant difference could be that students did not realize the differences between the graphs (Biesinger & Crippen, 2010). In our study, the threat for treatment validity was minimized by the inclusion of bullet points explaining the affordances of the dashboards and video tutorials sent to the participants in each treatment group at the beginning of the semester explaining the information displayed in the dashboards. Further research is needed to provide design guidelines that will influence learners’ cognitive outcomes in anxiety-inducive courses such as statistics and chemistry.

Conclusions

We caution against premature generalizations due to the inherent exploratory nature (Institute of Education Sciences & National Science Foundation, 2013) of learning analytics dashboard studies, the limitations inherent to the convenience sample used in this study (Etikan et al., 2016), and the lack of more nuanced behavioral data (e.g., daily or weekly uses of the dashboards) as factors in the models. Despite these limitations, our results are supported by the following methodological features: (a) application of an experimental research design; (b) authentic learning setting; (c) semester-long implementations, and (d) use of quantitative and qualitative data. As in many experimental studies, our sample was restricted to students from a single statistics course in a single university. Nevertheless, while more studies in different courses, universities, and geographic regions are necessary to investigate whether the results we found can be replicated in other contexts and with different groups of learners (e.g., K-12, undergraduate), the methodological aspects listed above make our studies’ findings and implications more transferable to other similar contexts and offer an important contribution to theories of motivation within the realm of learning analytics (Bodily & Verbert, 2017).

Taken together, our results suggest that the combination of performance and mastery learning components used in the predictive dashboard had a positive influence on learners’ interpretation anxiety by reducing it in comparison to the control group. The predictive dashboard also supported intrinsic motivation for those who started the course with higher levels of intrinsic motivation; however, it had a detrimental influence on motivation among less motivated learners, which challenges the assumption that self-referenced approaches are necessarily more motivational as suggested by previous studies (Jonathan et al., 2017; Martin & Elliot, 2016). Importantly, both the descriptive and predictive dashboards reduced negative attitudes towards statistics (worth of statistics anxiety), especially among learners who started the course with higher levels of aversion towards statistics. The reduction of negative attitudes towards statistics is a highly desirable outcome because, despite its practical applications, statistics is perceived as a difficult subject by many learners (DeVaney, 2016; Hanna et al., 2008). Furthermore, peoples’ positive perceptions may influence their motivation and actions related to a given domain (Valle et al., 2020).

These results support the growing body of literature on multiple goal orientation (Harackiewicz et al., 2002) by providing evidence that the combination of performance and mastery elements can support learners’ intrinsic motivation among highly motivated learners, can reduce interpretation anxiety, and can also reduce negative attitudes towards statistics particularly among more anxious learners (Cruise et al., 1985) in comparison to the control group.

Implications and future directions

Findings from this study provide important research-based guidance built on contextually rich lessons learned (Ritzhaupt et al., 2020; Valle et al., 2018) to support designers and instructors of highly anxiety-inducive courses such as statistics. More specifically, we found that:

Predictive dashboards featuring self-referenced elements may help highly motivated learners to sustain their level of motivation in the course. However, they may decrease the motivation of less motivated learners. Future investigations could explore whether predictive dashboards can be optimized by the inclusion of more contextualized performance-oriented information such as class average scores.

Predictive dashboards are more likely to reduce interpretation anxiety possibly because of the self-referenced elements used, which contextualized learners’ individual progression in the course without comparisons with their class. Future research could explore whether the inclusion of other theory-based elements such as motivation scaffolding prompts would also help reduce learners’ interpretation anxiety.

Predictive and descriptive dashboards combining performance and mastery goal orientation elements can be used to reduce learners’ negative attitudes towards statistics courses. Future studies could investigate whether this phenomenon is replicable in other STEM-related disciplines such as physics or chemistry.

To conclude, we hope that the theory-based approach we used in the conceptualization, development, and implementation of learner-centered learning analytics dashboards will motivate other researchers to explore the potential of the multiple goal orientation perspective to support learner’s motivation in anxiety-driven courses such as statistics (Dela Rosa & Bernardo, 2013; Harackiewicz et al., 2002; Pintrich, 2000a). Only when more theory-based learning analytics studies focusing on learners cognitive and affective outcomes are conducted will we be able to contribute to educators’ perceived utility of adopting learning analytics tools to support their pedagogical practices and their learners (Ali et al., 2013). Because, if “learning analytics are about learning” (Gašević et al., 2015, p. 64), learning analytics dashboards should be about learners!

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 48 kb)