Perceived Level of Usability as an Evaluation Metric in Adaptive E-learning: A Case Study with Dyslexic Children.

The global spread of COVID-19 has shifted the learning process towards e-learning. In this context, a critical challenge for researchers is to understand and evaluate the effectiveness of e-learning, especially when the learning is adapted to the needs of individual users. In this work we argue that the learner's perception of the level of usability of a system is a valuable metric that gives an insight into the learners' engagement and motivation to learn. Little attention has been paid to this metric. In this paper we explore why this is important and valuable. We present a case study which uses the System Usability Scale (SUS) questionnaire to measure the user's perception of usability as an indirect (proxy) measure of engagement. A between-subject experiment was conducted with 41 learners with dyslexia. The intervention group used the adaptive version of the e-learning system that matched the material to the needs of the learner. The control group used a standard version. At the end, learning gain and SUS scores were assessed. The correlation between learning performance and the perceived level of usability was positive and moderate (0.517, p < 0.05) among participants in the intervention group. However, learning performance and perceived level of usability were unrelated in the control group (- 0.364, p > 0.05). From this, and other work, it appears that using a learner's assessment of the usability of a system is an effective way to measure their attitude to their learning. It reflects their perception of its suitability to their needs and this, in turn, is likely to affect their engagement and motivation. As such, this provides an effective instrument to judge whether adaptation based on learner needs has been successful.

Introduction

Learning has changed dramatically recently, especially with the novel coronavirus (COVID-19) pandemic, and electronic learning (e-learning) has seen a substantial rise. The current challenge of e-learning is not only to make resources available to everyone, at any time and place, and in different formats, but also to offer learning properly—where the learner’s motivation, needs, characteristics and background are taken into consideration [42]. These factors highlight a new paradigm as a solution to the limitations of traditional e-learning [66], which does not provide personalization of learning content, interactivity and real-time execution. This new paradigm is called adaptive e-learning. Adaptive e-learning systems can solve these issues by altering the structure, content and presentation of the material to suit the different needs of the different learners [63].

Evaluating the effectiveness of adaptive e-learning systems is a topic that has received attention from many researchers, and a number of studies have identified success factors to maximize the benefits of these systems. Most of the research has focussed on the technical success of these e-learning systems and how learners’ characteristics affect the teaching [23]. At the same time, the technology has become more accessible and reliable [2]. More recent studies have targeted how learners’ attitudes and interactions are significant factors in success. An evaluation of these systems’ ability to meet learners’ needs and enabling their continuing education is therefore needed.

The learning experiences and achievements of learners is one challenge in the e-learning domain [55]. Learners’ satisfaction and learning outcomes are good indicators of the effectiveness of these systems [58]. Another essential indicator of e-learning quality is learners’ engagement. Learner engagement refers to the learner’s ongoing effort to support their psychological commitment to be motivated, acquire knowledge and achieve learning goals [26]. An increase in the engagement level of learners may serve as a strong predictor of improved performance and achievement [19].

Many approaches are used to measure engagement, such as self-report, interviews, observation and teacher ratings [30]. However, these direct methods can be challenging to implement without affecting the results and they require substantial time and resources. There is therefore a need for indirect measurements to assess engagement efficiently.

Another significant indicator of e-learning quality is usability [37]. Usability refers to the extent to which a product can be used by specified users to achieve specified goals with effectiveness and efficiency. If a learner perceives a system as easy to use, they will be more satisfied and engaged in using the system [12, 68], which in turn helps them achieve learning goals and improve their learning outcome. A highly usable, adaptive e-learning system improves learning performance by enabling learners to concentrate on learning tasks instead of focussing on the system’s functionality [51].

Although the level of usability has an impact on a learner’s motivation and engagement with their learning [12, 68], the feasibility of using perceived level of usability as a metric to evaluate the effectiveness of adaptation has received little attention [52] (especially in the context of learners with dyslexia—who are the target group in this case study). Without engagement with the learning content, it is difficult to enhance the learning performance of learners with dyslexia [36]. Therefore, assessing these learners’ engagement is necessary, in its own right. Also, there are theoretical models and reliable diagnostic tools to identify different characteristics and reading problems of individuals with dyslexia [11, 31]. Therefore, it is appropriate to address these different needs of dyslexia as a basis for adaptation to enhance their learning. In addition, usability research on e-learning largely ignores the learner’s perspectives [57], even though the learners are key users of e-learning environments and their characteristics and needs must be the main focus of usability research [12]. So, beyond assessing actual usability, we argue that the learner’s assessment (perception) of usability is affected by their overall experience. Therefore, for example, two systems with identical interfaces may be assessed differently, if the content of one is more successfully adapted to the needs of the learner. That is, we can use the learner’s usability assessment as a metric to assess the success of the adaptation. This, effectively gives us an indirect way to evaluate their satisfaction and engagement or the value that they attach to the learning.

This research aims to: (i) demonstrate that perceived level of usability is a significant indicator of the effectiveness of adaptation in e-learning systems as an indirect method to measure engagement. We do this by conducting an experimental study of learners with difficulties to understand whether perceived usability is related to learning performance when learning material is adapted to their needs; and (ii) examine a possible correlation between perceived level of usability and grades when the learning material is adapted to learners’ needs.

The findings of this research contribute to existing knowledge through providing insight into the viability of using ‘perceived level of usability’ in evaluating the quality of adaptation in e-learning systems by assessing learners’ engagement with the learning content indirectly. This adds novelty and originality to this research in the sense of the metric and methodology used to evaluate this dimension. The approach is implemented in an e-learning system to train learners with dyslexia (the target domain) and to evaluate the benefits of adaptation. It is observed through this research that learning performance is correlated to perceived level of usability when the material is adapted to learners’ needs. This implies that when a learner perceives the system as easy to use, they will be motivated and engaged, which in turn enhances learning performance.

This paper is structured in the following way: “Related Work” overviews the background on dyslexia (the domain in which this research is focused), adaptive e-learning systems and different techniques to evaluate the effectiveness of these systems. In “Case Study”, the proposed methodology to evaluate this research’s aim is described. Next, “Results” presents the results, followed by a detailed discussion of this research in “Discussion”. Finally, the conclusion and future research directions are presented in “Conclusions and Future Work”.

Related Work

Evaluating the effectiveness of adaptive e-learning systems is critical work, as the use of these technologies has skyrocketed during the COVID-19 pandemic. The literature review in this section covers three aspects: the definition of dyslexia and the difficulties it creates for learning, adaptive e-learning, and techniques used for the evaluation of e-learning systems.

Dyslexia

Dyslexia, a language-based learning disability, is the most common of all specific developmental disorders, affecting around 5–17% of the population [32]. The formal definition of dyslexia, according to the International Dyslexia Association (IDA) in 1994, is: “a specific learning disability that is neurobiological in origin. It is characterized by difficulties with accurate and/or fluent word recognition and by poor spelling and decoding abilities. These difficulties typically result from a deficit in the phonological component of language processing that is often unexpected in relation to other cognitive abilities and the provision of effective classroom instruction. Secondary consequences may include problems in reading comprehension and reduced reading experience that can impede growth of vocabulary and background knowledge” (p. 2) [45]. The components of this definition are illustrated in Fig. 1.

Fig. 1

Components of dyslexia definition adopted by the IDA [29]

According to this definition, the major academic skill deficit characterizing children with dyslexia is phonological processing difficulty, including problems with spelling, accuracy and/or fluency of single-word decoding and writing [29]. Reading comprehension can be affected as well [45]. However, the extent to which these deficits occur varies depending on the language’s orthography [1]. For example, dyslexia in Arabic is different from dyslexia in English [27]. Two types of languages exist: transparent and non-transparent orthographic languages. Readers in transparent orthographic languages, such as Italian and Spanish, face fewer reading difficulties than readers in non-transparent languages such as French and English [62]. Some languages, such as Arabic and Hebrew, feature two orthographies. Arabic is a widely spoken language with a considerable rate of dyslexia [15]; however, little research has targeted dyslexia in this language, and most of these studies focus on the complexity of Arabic rather than on targeting the needs of learners with dyslexia in the e-learning environment to enhance their experience. This research targets dyslexia in Arabic.

Adaptive  E-learning

E-learning has a significant impact on learners and has gained acceptance in all academic institutions. E-learning can be defined as “an innovative web-based system based on digital technologies and other forms of educational materials whose primary goal is to provide students with a personalised, learner-centered, open, enjoyable and interactive learning environment supporting and enhancing the learning processes” (p. 95) [57].

In contrast to traditional learning models, the e-learning environment supports the educational process for heterogeneous learners anytime and anywhere [22]. However, traditional, standalone e-learning systems nevertheless have drawbacks when compared to real-life teaching in classrooms, such as the lack of flexible tutorial support, presentation and feedback and the lack of adaptive support [66]. Thus, the e-learning field has shifted to adaptive e-learning, which can adapt learning material according to different learners’ requirements [63].

In the e-learning context, adaptation can be defined as a procedure for tailoring the learning environment to accommodate differences between learners [17] to improve learning performance [48]. The objective of adaptation differs from one system to another. Some adaptive e-learning systems aim to enhance the outcome of learners and increase their progress [4], while other systems have different objectives, such as increasing learners’ engagement [35], satisfaction and usability [9, 10].

Adaptive e-learning systems can provide suitable learning content according to different learners’ characteristics, such as previous knowledge level [21], learning styles [61], and personality [35]. Learners with dyslexia, like normal learners, differ in their needs and characteristics. These differences should be considered individually, instead of treating everyone in the same manner [3]. Among these characteristics of dyslexia are dyslexia type [4] and learning styles [14]. This research targets reading skill level of dyslexia as one characteristic in an adaptive e-learning system [5].

Techniques to Evaluate the Effectiveness of Adaptive E-learning

The prevalence of e-learning highlights the need to evaluate these systems’ effectiveness. The evaluation of these systems can be approached from a technical perspective [38] and from a user-centred perspective, as users are the main target of these systems—“the key elements in an e-learning intervention”. Thus, learners’ needs and abilities must be the main focus of adaptive e-learning when evaluating its effectiveness.

Different evaluation metrics exist. Among these metric are learning gain, learner satisfaction and perceived level of usability. The following paragraphs explain these metrics in detail.

Learning gain Learning effectiveness is a fundamental factor that should be measured when evaluating adaptive e-learning systems [53]. Learning gain, measured by pre- and post-tests, is a commonly used term to describe any improvement in the learner’s knowledge after a specific intervention [34]. Typically, to assess learning gain, the pre-test is administered to a group of learners based on their level of knowledge. Two balanced groups of learners are formed: one group that interacts with the non-adaptive version of the system and the other group that interacts with the adaptive version. If the second group scores higher (in terms of learning gain) than the first group in the post-test, the adaptation is considered a success [34].

For learners with dyslexia, it is crucial to assess the effectiveness of adaptive e-learning systems based on their needs by assessing learning gain. Previous research has demonstrated the success of adaptive e-learning systems in improving dyslexia learning gain after intervention, as seen in [4, 6]. In addition, assessing whether content learned can be generalised to a new context is a critical outcome in education [50]. This reflects the ability of learners to apply what they have learned in the intervention to analyse new material that is unfamiliar to them [44]. However, knowledge generalisation is rarely assessed in most research [41]. Therefore, a measure of generalisation to assess whether learners generalise and can apply learned content to new content is another metric for evaluating the effectiveness of adaptive e-learning systems. Generalisation can be assessed by using material in the pre- and post-tests that varies systematically in terms of similarity to the material used in the e-learning system [25].

Learner satisfaction Learner satisfaction is a critical element of the quality of e-learning experiences [43], and is considered a valid predictor of a system’s effectiveness [33]. Satisfaction refers to the pleasure that a user feels when they perform an action or receive something needed to perform an action [59]. In the Human-Computer Interaction (HCI) field, user satisfaction is visualised as the expression gained from interaction with a system [59]. Learner satisfaction is affected by several factors such as engagement and motivation to interact with the system. It is also associated with the extent to which they believe the system they interact with matches their needs [59].

The post-test questionnaire is used to assess learner satisfaction by asking learners questions about their satisfaction after using a system [34]. Several tools can be used to measure learner satisfaction, such as those of [40, 46]. Among these tools, the E-Learner Satisfaction (ELS) tool [65] is a commonly used tool to assess learner satisfaction in adaptive e-learning [9]. ELS is a reliable and validated questionnaire, consisting of 17 items related to four different factors: learning content, system interface, learning community and personalisation [65]. These items have 7-point Likert scales ranging from “strongly disagree” to “strongly agree” [65]. ELS measures overall satisfaction in addition to satisfaction related to each of the four factors [65]. ELS can be adapted to fit a specific research need [65], and is therefore widely used for a variety of e-learning systems [59].

For children, presenting the Likert scale from “strongly disagree” to “strongly agree” is not useful because of the difficulty they encounter in understanding and interpreting it [56]. One widely used alternative instrument is the Smileyometer [56], shown in Fig. 2. It uses pictorial representations based on a 1–5 Likert scale, or a 1-7 Likert scale as in [49], that enables children to identify their feelings by choosing one face. It can be adapted for any tool [60]. Moreover, it is easy to use, quick to complete and does not require any writing.

Fig. 2

A Smileyometer [56]

For learners with dyslexia, measuring the degree to which they are satisfied with their assigned version of the system is useful. It allows them to indirectly measure whether they perceive the system as suitable for their needs. If the system meets their needs and characteristics, they will be more satisfied and motivated, which in turn affects learning effectiveness [3].

Perceived level of usability The way end users perceive their interaction with systems is a significant aspect of e-learning that can lead to a better user experience. Usability is defined as the “extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” [39].

Perceived level of usability can be assessed by asking end users to assess the system’s usability after use—using standardised usability questionnaires, as in [28]. A commonly used instrument for evaluating the perceived usability of systems in HCI is the System Usability Scale (SUS) questionnaire [16, 37]. SUS has a high degree of validity and reliability [13] and is a valid instrument for comparing the usability of two or more systems [54]. Furthermore, it can be adapted for different contexts [54]. When using SUS, reliable results are evident even with small samples [64]. It can be adapted to be used by children [4] in addition to being used as a usability tool for Arabic users with a high degree of reliability [7].

The SUS includes 10 mixed-tone items on a five-point Likert scale ranging from strongly disagree (1) to strongly agree (5). The even numbered items have a negative tone and the odd numbered items have a positive tone. To calculate the overall SUS score, the rules are as follows: for each even numbered item, the respondent’s answer scale is reduced by 5, and for each odd numbered statement, the respondent’s answer scale is reduced by 1. The overall SUS value of each respondent can be obtained by multiplying the sum of scores by 2.5.

Good learning performance and achievement is associated with strong engagement with the learning content [19]. Hence, motivation to learn is used as a usability measure to evaluate an e-learning system [67]. In this research we evaluate the effectiveness of adaptation in the e-learning environment by assessing the viability of perceived level of usability as an indirect metric to measure engagement, and whether perceived level of usability is related to learners’ learning gain. We argue that this is an effective way to assess the benefit of adaptation. If a user perceives a system as being easy to use, this will reflect that they are more engaged and motivated, which indirectly improves their learning performance.

Case Study

This section details the methodology used in this research. An experimental study was conducted with Arabic learners with dyslexia in primary schools. We targeted this group for several reasons. First, previous research, such as [8, 18], has failed to consider the variations in skills and backgrounds among children with dyslexia in e-learning. As a result, learners report lower satisfaction and engagement [3]. Second, a number of studies utilised instructor evaluations of e-learning systems, ignoring the learners’ perspective [8, 18]. Third, little research has targeted dyslexia in Arabic speakers [4]. Finally, accessibility to children with dyslexia was more straightforward than those in intermediate or high schools.

A web-based e-learning system was designed and implemented to evaluate perceived usability with Arabic learners with dyslexia in primary schools. The system provided six training sessions, each with 20 different word reading/recognition activities. The material used in the system was derived from the school curriculum. Three fundamental Arabic reading skills were chosen that serve as basic foundations for advanced skills (according to interviews with experts). These reading skills are described fully in [5]:Figure 3 shows the material structure of the skills in the system.

Fig. 3

The training material structure in the e-learning system

Reading letters with short vowels (S1).

Reading words with Sakin letter(s) (S2).

Reading words with short vowels and Sakin letters (S3).

In each piece of reading skill material, the difficulty increases gradually. In S1, the activities begin with letters with a simple short vowel (fat-ha (/a/)), followed by (kasra (/i/)) to the advanced (dammah (/u/)). In S2, the activities begin simply (one Sakin letter at the end of two-letter words) to advanced (two Sakin letters in the middle or at the end of three- and four-letters words). A Sakin letter is a letter with a small circle on top of it indicating no vowel. In S3, the activities begin simply (three-letter words with fat-ha (/a/)) and become more advanced (three- and four-letters words with different short vowels and Sakin letters).

An example of a training activity from S3 is shown in Fig. 4. At the top of the screen, the activity question is presented. The target word is denoted as dots inside a rectangular box under the query with the corresponding audio beside it (a question mark icon). The learner can listen to the target word audio and then choose the correct word from among three choices, of which only one is correct. If the learner selects the correct answer, written and spoken praise are provided. Otherwise, negative written and spoken feedback is provided. Training progress is provided as a progress bar at the top of the screen. After every seven completed activities, a motivational message is presented to increase the learner’s confidence [15].

Fig. 4

A screenshot example of a training session of S3

Two versions of the system are developed to support the experimental conditions. The standard version fixes the material to cover all three reading skills, while the matched version matches the activities to the reading skill of each learner. If a learner already masters a skill, then the system will not cover the activities of this skill. Rather, it will provide the activities that need to be mastered by the learner. Both versions of the system are identical in layout; the only difference is the provided material in the activities to investigate whether adapting learning material affects learners’ perceived level of usability.

Hypotheses

One hypothesis was formulated: Perceived usability positively correlates to learning gain when learning material is adapted to learners’ needs.

The independent variables were the standard and matched versions of the e-learning system, while the dependent variables were learning gain and the perceived level of usability.

Measurements and Data Collection Tools

The reading skill level of learners was determined using three reliable reading skill level diagnostic tests [5]. These tests meet the requirements of the standardised tests for students with learning difficulties in the Kingdom of Saudi Arabia (KSA). The tests are: (1) reading letters with short vowels, (2) reading words with Sakin letters, and (3) reading words with short vowels and Sakin letters. Each subject was asked to read words in these tests aloud to determine their ability to read the words correctly.

Two measurements were used in this experiment: learning gain and perceived level of usability. Learning gain was assessed directly after finishing the experiment and a learning gain measure was derived (post-test score − pre-test score). These tests included different words from the curriculum and were validated by special education experts. Two types of learning gain were measured in this study. The first type was learning gain of seen words (words included in the training sessions) to evaluate the effectiveness of the adaptation. The second type was learning gain of unseen words (words that were not included in training sessions) to investigate the ability to generalise. The results of learning gain are reported in [6]. In this research, we used the results of learning gain of seen words to investigate the correlation with the perceived level of usability.

Perceived level of usability was measured using the SUS tool, replacing the 1–5 Likert-scale with the Smileyometer. The results of perceived level of usability are reported in [5].

Procedure

The process of this research was conducted in accordance with institutional ethical policy. Signed consent forms from subjects’ parents/guardians and primary schools were obtained.

The experiment procedure is illustrated in Fig. 5. In one session, subjects were welcomed and introduced to the experiment’s process. Next, their demographic information (grade, age) was collected and diagnostic tests, including the pre-test, were administered. Then, all subjects were randomly assigned to the balanced control or intervention group. The control group used the standard version of the learning system, while the intervention group used the matched version.

Fig. 5

Experiment procedure

The training task was completed in a quiet room inside each subject’s school for three weeks during two sessions per week (a total of 6 sessions). Each session lasted approximately 30 min. Each subject worked individually with the system without knowing which experimental condition they had been allocated.

At the end of all the training sessions, subjects were immediately directed to complete the post-test followed by the SUS tool.

Subjects

Participants who were native speakers of Arabic, officially diagnosed with dyslexia from different primary schools in Jeddah city, KSA participated in this experiment. The sample (n = 41) was split into two subgroups, matched for age, grade, reading skill level and prior reading accuracy (from the pre-test) and randomly assigned either to the intervention or the control condition (see Table 1). All subjects had previous experience using electronic devices. Due to the separation of boys and girls in KSA educational institutions, researchers did not have access to male learners. This had the advantage of reducing variance between subjects.

Table 1

Subjects’ distribution in terms of grade, age, reading skill level and pre-test

Group	Control group (N = 21) Mean (SD)	Intervention group (N = 20) Mean (SD)	Groups comparison
Female	21	20
Second grade	10	11
Third grade	6	4
Fourth grade	5	5
7 Years-old	4	5
8 Years-old	7	6
9 Years-old	6	5
10 Years-old	4	4
Reading skill level (S1)	3	3
Reading skill level (S2)	5	6
Reading skill level (S3)	13	11
Reading accuracy (pre-test)	4.19 (1.99)	3.9 (1.74)	U = 196, Z = 0.371, p = 0.711
Reading accuracy (learning gain)	1.57 (1.80)	4.6 (1.67)	U = 46.5, Z = 4.307, p = 0.000017

Results

The results were analysed using IBM SPSS Statistical package (version 27).

To assess the relationship between learning gain and perceived level of usability for both groups, a Spearman’s correlation was run. As shown in Fig. 6, there was a moderate, positive correlation between learning gain and perceived usability among subjects in the intervention group (). In contrast, perceived usability and learning gain were unrelated among subjects in the control group ( ). Therefore, perceived usability was associated with greater learning gain among learners using the matched version of e-learning.

Fig. 6

Scatter-plot of learning gain by SUS overall score by groups

Further investigation was conducted to determine which factors are related to SUS score in each group (i.e., learners’ grades), as shown in Fig. 7. In general, among subjects in the intervention group, specifically in Grade 2, there was a moderate but statistically significant positive correlation between perceived usability and learning gain, . However, there was no statistically significant correlation among subjects in Grade 3 () and Grade 4 (). In contrast, usability and learning gain were unrelated among subjects in the control group in all grades, Grade 2 (), Grade 3 () and Grade 4 ().

Fig. 7

Scatter-plot of learning gain by SUS score by grade for the control and intervention groups

Discussion

The COVID-19 pandemic has forced educational institutions to switch to e-learning [47], making it necessary to evaluate the quality of e-learning systems. A learner’s degree of engagement is affected by the learning system’s usability, which impacts the learner’s motivation to engage in the material [20].

This research aims to fill a gap in the literature by assessing the viability of using the perceived level of usability when evaluating an adaptive e-learning system from the learners’ perspective. This research contributes to the field of adaptive e-learning systems evaluation by revealing the significance of the learner’s perception of the usability. Our argument is that the learner will (perhaps not consciously) recognise that their learning is matched to their needs. They will, therefore attach more value to the learning and this will be reflected in the usability scores that they award. This is an effective way to assess whether the adaptation has been effective when two, otherwise identical versions of a system are compared.

These research results indicate that perceived level of usability is correlated to learners’ learning gain when the learning material is adapted to their needs. This means that if learners perceive an e-learning system as more usable, their engagement is enhanced, which in turn affects their learning performance. This finding is in line with prior research [5, 10].

Perceived level of usability can also, therefore, be used as an indirect metric for measuring learner engagement. This is parallel to a previous study [35], that measured dropout rate as a proxy for engagement indirectly. This means that learners who are more engaged will use the e-learning system for longer and that this, in turn, leads to improved learning outcomes.

These research results also confirm previous study findings that showed that perceived level of usability significantly correlates with learners’ engagement [24]. Learners who perceive the system as easy to use, will be more engaged and motivated.

Furthermore, the results show that perceived level of usability is a key element in evaluating the effectiveness of adaptive e-learning. This confirms previous research findings that showed that a highly usable e-learning system allows learners to focus on the learning process rather than on the functionality of the system [51].

Conclusions and Future Work

This research focussed on evaluating an adaptive e-learning system using perceived level of usability to predict learners’ engagement. This work adds to existing knowledge through new insights into perceived level of usability as a significant metric for evaluating these systems’ quality by indirectly assessing their engagement and how it can relate to their learning performance.

In future research, it would be useful to conduct a study with other subjects to evaluate the correlation between perceived usability and learning gain, for instance high school and undergraduate learners. Furthermore, it would be useful to investigate whether there is any differentiation in the perceived level of usability in other learning subjects (i.e., math and science). In addition, it would be useful to examine possible relationships between SUS score and age, and male learners.