Use of an Adaptive e-Learning Platform as a Formative Assessment Tool in the Cardiovascular System Course Component of an MBBS Programme.

BACKGROUND: Technology-enhanced learning includes the adaptive e-learning platform, a data-driven method with computer algorithms, providing customised learning enhancing critical thinking of individual learners. "Firecracker" - an online adaptive e-learning platform, and assessment software, promotes critical thinking, helps prepare students for courses and high-stakes examinations, and evaluates progress relative to co-learners. The objectives of this study were to determine the usage rates of Firecracker, examine the performance of Firecracker formative quizzes, identify the correlation between Firecracker use and performance with that of performance at summative course assessments, and assess students' satisfaction with Firecracker usage.
METHODS: Study participants were Year-2 MBBS (Bachelor of Medicine, Bachelor of Surgery) students (n=91) of the Faculty of Medical Sciences, The University of the West Indies, Cave Hill Campus, Barbados. The Firecracker Administrator uploaded quizzes covering basic science content in the Cardiovascular System course. Access, usage, and performance on Firecracker formative quizzes were retrieved from the Firecracker dashboard. A questionnaire sought the views of study participants.
RESULTS: Seven sets of quizzes were administered over nine weeks, with weekly student completion rates ranging from 53% to 73%. Mean quiz scores ranged from 52% to 72%. Students completing >4 quiz sessions compared to those completing ≤4 demonstrated significantly better performance in Firecracker quizzes (P<0.01), final examinations (P<0.01) and in-course assessment plus final examination (P<0.05) scores. Correlations between overall Firecracker performance and in-course assessment marks (P<0.05); between total overall Firecracker performance and final examination (P<0.01); and overall Firecracker performance and total course marks (P<0.01) were all significant. Most students (70%) were happy using Firecracker and felt it complemented coursework (78%) and prepared them for course exams (58%) (P<0.01).
CONCLUSION: Overall, Firecracker was perceived very positively and welcomed by the students. Students were satisfied with the Firecracker as a formative assessment tool, and its use correlated with improved performance in the course examinations.

Background

Technology-enhanced learning has become popular and now plays a vital role in teaching, learning, and assessment in medical education.1–3 It supports and supplements teaching to deliver a learner-specific, personalised and adaptive learning environment according to individual learner needs through real-time experience.1,4 This experience is enhanced through timely feedback, using different modes to deliver course materials, availability and access to various resources and adapting the pace and path of learning based on preference and performance of students.1,4 The most widely used e-learning platforms in medical education have non-adaptive e-learning environments (NEE), which provide standardised training for all students.5,6 Though NEEs provide interactivity, feedback, and practice exercises, they fail to consider learners’ characteristics and provide personalised education and training.6

On the other hand, adaptive e-learning environments (AEEs) potentially increase learning efficacy and efficiency by building individual student profiles and using simple adaptive techniques to provide a personalised learning experience.7,8 AEEs have been commonly used in mathematics, physics and related disciplines for learning factual knowledge and skill development.9 Sharma et al (2017) defined adaptive learning as a “process that provides an individualised learning experience with technologies designed to determine a learner’s strengths and weaknesses.”1 Adaptive learning, also known as adaptive teaching, is a data-driven method with computer algorithms that provides customised learning to engage individual learners to enhance their learning.9

Adaptive learning utilises techniques to implement a variety of adaptivity: (i) designed adaptivity, where the educator designs the instructions so that the learner can master the content; and (ii) algorithmic adaptivity, which uses algorithms to determine the extent of the learner’s knowledge; and the best sequence of instructions, or training strategies to provide a personalised learning experience to guide toward content mastery.9 AEEs analyse how individual learners interact with courseware and perform and then predict the kinds of contents and resources that meet their needs, including where necessary specific remedial activities.10 As a result, adaptive learning can optimise learning efficiency.11 A recent systematic review and meta-analysis demonstrated that AEEs appear to be effective in improving skills in both professionals and students.9 Adaptive learning can be accurate and fluent and helps guide the spacing and sequence of learning events. Adaptive learning technology with a spacing element, such as Adaptive Response Time-based Sequencing (ARTS), has been shown to optimise learning and retention.12 Other studies have also reported improved learning with adaptive spaced education.13–15 Different adaptive e-learning platforms exist in the medical education arena such as Smart Sparrow Adaptive eLearning Platform, Articulate Storyline 360,16 and Firecracker ().17

Recently medical education across the world has experienced a significant disruptive change because of the COVID-19 pandemic, and adaptive learning platforms have been rapidly and innovatively utilised to deliver teaching remotely by medical schools.18,19 It is highly likely that the use of emerging technology in medical schools will play a crucial role to enhance teaching and learning even after the resolution of the pandemic.

Firecracker, an online adaptive learning platform and assessment software, helps prepare students for their courses and high-stakes exams and determines their progress relative to peers. Adapted Spaced Learning is used as a fundamental technique in Firecracker’s adaptive learning platform, and the platform also utilises several other well-established principles of learning to improve students’ academic performance and long-term retention of memory ().20 A recent scoping review conducted by Versteeg et al (2020)21 showed that online spaced learning had been widely used in health profession education.13,22–26 Based on the findings, the authors proposed the following comprehensive definition of spaced learning:

Spaced learning involves [specified] educational encounters that are devoted to the same [specified] material, and distributed over a [specified] number of periods separated by a [specified] interstudy interval, with a [specified] learning outcome after a [specified] retention interval21

The objectives of this study were to: (a) determine the usage rates of Firecracker, (b) examine the performance of Firecracker formative quizzes, (c) identify the correlation between Firecracker use and performance with that of performance at summative course assessments, and (d) assess student satisfaction with Firecracker usage.

Method

Ethical Considerations

Ethical approval was obtained from the Institutional Review Board, The University of the West Indies, Faculty of Medical Sciences, Cave Hill Campus, Barbados (IRB No. 180405-B).

Context

The Faculty of Medical Sciences (FMS), Cave Hill Campus, the University of the West Indies (UWI), Barbados has a 5-year undergraduate Bachelor of Medicine, Bachelor of Surgery (MBBS) programme. Students are accepted into the programme either immediately after the completion of high school board examination or after the completion of at least a - degree in science. Students are from the English speaking Caribbean countries, and most remain and work in the region. The first three years of the programme is largely preclinical with an integrated system-based design to the curriculum. There are continuous in-course (40%), and final summative assessments (60%).

Study Design, Setting and Participants

This study was conducted in 2018, of the academic year 2017–2018, on the second-year medical students enrolled in the MBBS degree program at the FMS, UWI, Cave Hill Campus, Barbados. Firecracker was introduced in the 2017/2018 academic year to supplement the teaching of the Cardiovascular System (Course code: MDSC 2103) course, a year-2 MBBS course that runs over 13 weeks of the first semester. Second-year students who were taking the Cardiovascular System course were enrolled in Firecracker. The Cardiovascular Course consists of lectures, seminars, case/problem-based learning, tutorials, demonstrations and laboratory practicals, use of multimedia, and the University’s E-learning course management system. Grades for the courses are based on a combination of continuous assessments [practical and multiple-choice questions (MCQs)] and course final exam consisting of MCQs only. The course has one In-Course Examination, carrying 25% of the total marks, and one Final Examination that represents 60% of the total. The laboratory practicals in anatomy, physiology, pharmacology, and microbiology, and seminars delivered by the students constitute the 15% of the total. The In-Course Examination has 40 MCQs to be answered in 60 minutes whereas the duration of the Final Examination is 2 hours and students have to answer 90 MCQs.

Firecracker had a well-established questions bank. Questions and quizzes were uploaded weekly for 9 weeks by the Firecracker Administration. The questions and quizzes were grouped into three major types: analytical, problem solving, and knowledge-based, and covered all the major areas of the cardiovascular system. In a set of 20 questions, 4 to 6 questions were knowledge-based, 8 to 14 questions were analytical, and 3 to 6 were problem solving. The questions for the Cardiovascular System course were developed based on: (a) structural basis (both micro and macro) and structure–function relationship of different levels of organization of the human body; (b) reports of abnormal cases and parameters in patients; (c) reports of normal cases and measurements of normal variables in healthy individuals; (d) measurement of variables of the body under stressful conditions; (e) description of cases under hypothetical set-up or experimental conditions; (f) analysis and description of graphical presentation of data; and (g) classification and possible mechanism of actions of external agents on body functions. Firecracker generates new contents every week and continuously reviews item characteristics with the appropriate level of difficulty. Students received a daily assignment of review questions covering topics relevant to their studies and prioritized by Firecracker’s learning algorithm. Students also received a relevant daily patient case (clinical vignette). Staff received full access to the Firecracker platform allowed them to track student progress and identify at-risk students.

Data Collection

The access, usage, and performance at Firecracker formative quizzes were retrieved from the Firecracker dashboard. A paper-based questionnaire, administered in Week 11 of the semester, was used to seek the views of students who participated in Firecracker (). A printed cover letter providing a brief description of the study and a request of consent was attached to the questionnaire. Those who consented to participate in this study signed the informed consent form and filled out the questionnaire.

Statistical Analyses

Data from the paper-based questionnaires were entered into Microsoft Excel Database. The t-test was used to determine the significance of differences in mean scores of two groups, correlation (r) was used to explore the relationship between Firecracker quiz performance with in-course, final examination and total course marks, and chi-square (χ2) to determine the degree of agreement of feedback among the respondents were performed using SPSS statistical software version 24. Results were considered significant if the p-value was <0.05 and <0.01. Further, Cohen’s d test was used to determine the effect size.

Results

Firecracker Statistics

Ninety-one students of the second year MBBS programme enrolled in Firecracker and seven sets of quizzes were administered in the Cardiovascular System course. Overall, students received 14,089 review questions organized into 1187 topics and 19 subjects, as well as 2003 clinical vignette questions. The number of the questions received weekly were: Week 1:11; weeks 2 through 5: 20/each week; weeks 6 and 7:20, and weeks 8 and 9:20, making a total of 131 questions. The percentages of the students who completed the quizzes were 73%, 68%, 62%, 56%, and 58% in weeks 1 through 5, and 58% and 56% in weeks 6–7 and 8–9, respectively. The weekly student completion rate of the quizzes ranged from 53% to 73%. The weekly average scores of the students were 58%, 56%, 55%, 53%, 55%, in weeks 1 through 5, and 72% and 52% in weeks 6–7 and 8–9, respectively. The mean scores achieved on the quizzes during this period ranged from 52% to 72%. Four students were listed as “At Risk” throughout the entire course, indicating that they answered ≤35% questions correctly. The Firecracker team had 9 weekly online meetings with the course coordinator.

Academic Performance and Firecracker Usage

Statistically significant differences were noted between those who completed four or fewer quiz sessions versus those who completed more than four quiz sessions in relation to the performance in Firecracker quiz assessment (P<0.01), final examinations (P<0.01) and total marks (P<0.05) (Table 1). Further, Cohen’s d effect size revealed that completion of more quizzes had a significant effect on student performance in all the above categories of assessment.

Table 1

Differences of Performance in Different Forms of Assessment Between Those Who Completed 4 or Less Quizzes vs More Than Four Quizzes

	Quizzes	N	Mean	Std. Deviation	t ratio⁑	P value	Cohen’s d†
Firecracker Quiz performance	4 or less Quizzes completed	19	1.60	0.70	11.25**	0.00	2.98
	More than 4 Quizzes completed	50	3.87	0.77
In-course assessment marks (40%)	4 or less Quizzes completed	19	23.86	5.52	0.987	0.327	0.27
	More than 4 Quizzes completed	50	25.32	5.44
Final Exams marks (60%)	4 or less Quizzes completed	19	34.31	4.60	2.78**	0.007	0.78
	More than 4 Quizzes completed	50	38.29	5.54
Total course marks***	4 or less Quizzes completed	19	58.17	7.92	2.12*	0.038	0.60
	More than 4 Quizzes completed	50	63.61	10.02

Notes: *Significant at the 0.05 level (2-tailed). **Significant at the 0.01 level (2-tailed). ***In-course assessment + Final marks. ⁑t-test = significant differences of performance in different forms of assessments between various classified subgroups. †Cohen’s d interpretation: 0.20 = “small” effect size, 0.50 “medium” effect size and 0.80 “large” effect size.

A low significant correlation between total Firecracker quiz marks and in-course assessment marks (P<0.05); a moderate significant correlation between total Firecracker quiz marks and final examination; and total Firecracker quiz marks and total course marks (P<0.01) was observed.

As shown in Table 2 statistically significant differences existed between students who participated and those who did not participate in relation to the performance in final examinations, in-course assessment and total course marks obtained (P<0.01). Further, Cohen’s d effect size revealed that participation of students in Firecracker had a significant effect on student performance in all the above categories of assessment.

Table 2

Differences of Performance in Different Forms of Assessment Between Students Who Participated vs Those Who Did Not

Marks	Category	N	Mean	Std. Deviation	t ratio⁑	P-value	Cohen’s d†
In-course assessment marks (40%)	No Firecracker	21	21.04	6.62	2.71**	0.008	0.64
	Firecracker	69	24.92	5.46
Final Exams marks (60%)	No Firecracker	21	30.47	8.96	4.15**	0.000	0.90
	Firecracker	69	37.19	5.56
Total course marks***	No Firecracker	21	51.51	14.01	3.92**	0.000	0.88
	Firecracker	69	62.11	9.74

Notes: **Significant at the 0.01 level (2-tailed). ***In-course assessment + Final marks. ⁑t-test = significant differences of performance of different forms of assessments between various classified subgroups. †Cohen’s d test - d=0.2 = “small” effect size, 0.5 “medium” effect size and 0.8= “large” effect size.

Students’ Feedback

Of the 91 students enrolled in Firecracker, 60 students completed the evaluation questionnaire (response rate 66%), which was administered in Week 11 of Semester One. Seventy percent of the students reported being happy using Firecracker, and the rest were neutral (26.7%) or unhappy (3.3%) (p=0.000). The majority of students found Firecracker helpful in complementing coursework (78.3%) (p=0.000) and preparing for course exams (58.4%) (p=0.008); used flashcard questions and topic summaries provided by Firecracker; and agreed that the weekly Firecracker quizzes were well-aligned with the materials assessed and information learned in the classes (68.3%) (p=0.000) (Table 3).

Table 3

Students’ Feedback on the Usage of Firecracker

	Strongly Disagree	Disagree	Neutral	Agree	Strongly Agree	One Sample χ2‡	df	Level of Sig
Used Firecracker to complement coursework and study	0 (0%)	6 (10%)	7 (11.7%)	20 (33.3%)	27 (45%)	20.93**	3	0.000
Used flashcard questions and topic summaries	3 (5%)	3 (5%)	10 (16.7%)	18 (30%)	26 (43.3%)	33.17**	4	0.000
Used Firecracker regularly	5 (8.3%)	12 (20%)	16 (26.7%)	17 (28.3%)	10 (16.7%)	7.83	4	0.098
Firecracker helped prepare course and final exams	3 (5%)	8 (13.3%)	14 (23.3%)	16 (26.7%)	19 (31.5%)	13.83**	4	0.008
Weekly quizzes assessed materials learned in class	4 (6.7%)	5 (8.3%)	10 (16.7%)	24 (40%)	17 (28.3%)	23.833**	4	0.000

Notes: **Significant at the 0.01 level (2-tailed). ‡Chi-square (χ2) was used to determine the degree of agreement about the feedback among the respondents.

Discussion

The main findings of this study include the following:

Students used Firecracker as a formative assessment tool (students completed 62% quizzes throughout nine weeks) and showed their satisfaction with adaptive learning technology.

Firecracker usage was found to be significantly associated with better examination performance.

Students who had completed more than 4 quiz sessions, secured better grades in Firecracker assessment, final examinations, and in-course assessment + final examination.

A significant correlation was observed between overall Firecracker performance and final examination grade, and overall Firecracker performance and total course grades (in-course assessment + final examination).

Students who participated in Firecracker, performed better both in the final examination, and in-course assessment.

Over the past several decades, higher education has evolved significantly, mainly driven by technological advancement, innovative curricular design, and diversity of pedagogical techniques.27,28 One of these techniques is adaptive spaced learning technology, which improves learning by attuning the level, spacing, and sequencing of learning events to each learner, allowing more efficient learning, better retention, and certification of mastery.10 Several meta-analyses demonstrated effective results of the efficacy of AEEs in comparison to large-group teaching among high school and university students.29–31

However, despite promising evidence of the efficiency of AEEs for knowledge acquisition and development of cognitive skills in higher education, their efficacy in improving learning outcomes in medical education has not yet been researched thoroughly. American Medical Association recommended that AEEs provide the opportunity to prepare today’s medical students for lifelong learning in the changing health care system (American Medical Association, 2019). Kellman and his colleagues conducted extensive research on adaptive learning technologies in higher education, including medicine and found remarkably promising results.10,32,33 Krasne et al (2013) used a perceptual and adaptive learning module (PALM) that utilised 261 unique images of cell injury, inflammation, neoplasia, or normal histology and showed evidence for improved recognition of histopathology patterns by medical students.33 Fontaine et al (2019) conducted the first systematic review and meta-analysis to evaluate the efficacy of adaptive e-learning in health professionals and found statistically significant improvements in learning outcomes in 12 out of 17 studies.9 In a study in the USA, Kerfoot et al (2010)13 demonstrated that adaptive spaced education significantly enhanced students’ retention of medical knowledge and improved composite end‐of‐year test scores (P<0.001). Another study found that implementation of a spaced education–based app study program in a third-year medical school surgery rotation was associated with higher scores on standardised tests in the National Board of Medical Examiners (NBME) examination.24 Our findings are consistent with the findings of these studies.

Our findings also demonstrated that students who participated in Firecracker performed better in in-course assessment and final examination. Bartsch (2016)34 demonstrated that Firecracker users (who answered at least 1500 flashcards in the Firecracker program prior to their exam) scored an average of 15 points higher in USMLE Step 1 than non-Firecracker users (who did not have a Firecracker account or answered fewer than 1500 flashcards in Firecracker prior to their exam). The average score (229.9 ±21.3) in Step 1 USMLE by non-Firecracker users was significantly (p <0.00001) lower than their Firecracker-user counterparts (245.0±16.6). The author also found that exposure to and mastery of flashcards within the Firecracker platform was correlated with significantly increased Step 1 scores. Bartsch (2016), in another study, examined the use of Firecracker with curriculum alignment to enhance student retention and summative exam performance and noted that Firecracker’s weekly formative assessments were significantly predictive of course summative exams.35 Moreover, student completion of weekly formative quizzes was also correlated with significantly increased summative final exam scores. We have also recorded that total Firecracker quiz grades were significantly correlated with final examination grades, and total Firecracker quiz grades and total course grades (in-course assessment and final examination). Other studies examining formative assessments in the form of practice quizzes in medical schools have also found predictive value for performances on summative exams.36–40

Participation of medical students in Firecracker was voluntary in the present study. Rashid et al (2017) observed an association between participation in voluntary practice quizzes and better performance on summative examinations.40 Non-participation was a common finding when these are voluntary exercises. In the current study, approximately 68% (ranging from 54% to 73%) of students participated in the weekly Firecracker quiz session, and 21% elected not to take part in any of these sessions. Other studies found even lower participation rates in similar voluntary mock tests, ranging between 40% and 70%.41–44 Rashid et al (2017)40 recorded 12% non-participation in their study, which is much lower than the findings from our research. Formative assessment and practice quizzes were found to be one of the least commonly used teaching and assessment modalities in medical education.45 The 2016 Report of the regional accreditation body, Caribbean Accreditation Authority for Education in Medicine and other Health Professions (CAAM-HP), highlighted the need for more formative assessments in our medicine programme. Hence, Firecracker was piloted to comply with CAAM-HP recommendations.

To summarise, formative self-testing resources and adaptive e-learning platforms are rapidly becoming an essential component of the medical curriculum. However, the effectiveness of these strategies needs to be evaluated in a variety of medical education settings and formats. Medical educators should integrate regular formative assessments into their curricula to improve student outcomes. Student participation in formative assessments gives instructors an understanding of knowledge gaps both for individual students and their instructional effectiveness. Firecracker’s adaptive system uses data collected from these formative assessments to help students quickly remediate their weak areas. Moreover, Firecracker’s adaptive flashcard system helped prepare students for their formative quizzes and final exams.

In addition to the limitation of non-participation as noted above, interpreting the effects of a practice quiz on summative performances of a course also has limitations, as other academic and non-academic variables may affect student performance. Besides, our study would have benefited from assessing student motivation and level of engagement in the course to identify the factors associated with participation in voluntary activities such as practice quizzes.

Conclusion

Firecracker was welcomed by most of the students, perceived to be helpful, and the use of this software was associated with better academic performance. Firecracker is likely to be useful in other courses for the continuous monitoring of students’ progress. Future studies should focus on examining why a significant number of students are reluctant to make use of the benefits of using Firecracker, or similar formative assessment tools, to their fullest and how these resources can be more constructively integrated in the curriculum to promote student performance and improve academic practices.