Attitudes of an international student cohort to the Quizlet study system employed in an advanced clinical health care review course.

Computer-based learning applications and mobile technology have transformed many aspects of the educational experience over the last decade, producing software aimed at improving learning efficiency and streamlining the presentation of course materials. One such class of software, purpose-created to take advantage of spaced learning and spaced testing principles, are electronic flashcard applications. We provide a perspective on the novel use of the Quizlet flashcard application in a tertiary educational setting. To reduce cognitive load for international graduate dental students taking a pharmacology review course, we implemented Quizlet, which integrates both spaced learning and self-testing, to improve the student learning experience. This study assessed students' perceptions of the Quizlet flashcard system in a student cohort comprised of two consecutive years' classes (n = 51 students in total). Results indicated broad acceptance of Quizlet based on ease of use of the software and ease of study of the material. Our data provide insight into the use of this common software in a professional healthcare tertiary education setting and further demonstrate the successful application of electronic flashcards for a mixed international student cohort. Further research should include an assessment of the impact of flashcard on long-term knowledge retention in this setting. Supplementary Information: The online version contains supplementary material available at 10.1007/s10639-022-11371-3.

Introduction

Learning should be intuitive, but for many students, it is not; even for those best able to learn, it is a process that can be enhanced by the way information is presented. The student learning experience can be influenced by factors at all levels, but over the last few years – and particularly over the recent pandemic period – the presentation format has perhaps been most influenced by the expansion of computer-based educational technologies. In the current study, we investigate student response to the implementation of a commercial computer-based learning application (app), Quizlet, in a higher education health science setting for an international student cohort. Based on the principles of spaced learning, Quizlet is an electronic flashcard app that uses retrieval practice, guidance fading, and goal setting.

Remarkably, despite thousands of publications including the term Quizlet, there seem to be none that examine the use of this learning application by course instructors for training in tertiary education for anything other than language-related learning. Although electronic flashcards are commonly used in healthcare education, mostly by students for self-study but also to a lesser degree as part of formal course structures, Quizlet – a widely used flashcard software – has almost exclusively been studied in primary and secondary education settings. Data for use of Quizlet in a tertiary healthcare setting is effectively absent. Here we describe the novel use of this widespread, distributed-learning, educational technology in a clinical education setting for international healthcare professionals.

We examine the literature relating to flashcard use in higher education, focusing on healthcare settings within the following context:

Spaced learning and retrieval practice

Cognitive load

Challenges to study met by international students

We further present data from a survey of student perceptions of Quizlet from an international student cohort.

Spaced learning

Spaced learning refers to the acquisition and retention of information using methods that allow either timed gaps between the study of individual information items within a data set – a spacing effect, lasting seconds – or timed gaps between the study of entire data sets – a gap effect, lasting hours to days (Toppino & Gerbier, 2014). Paradoxically, evidence consistently suggests that the longer the spacing or gap, the more effective the retention of knowledge. Despite evidence that spaced learning is effective, the evidence of its benefit is not wholly clear when compared to the commonly used student habit of massed practice.

Massed practice versus spaced practice

The magnitude of the efficacy of spaced practice is variable under test conditions, although compared to massed practice (i.e., cramming information) it has a consistently positive effect on student learning, particularly in complex subjects (Harbeson, 1989; Shebilske et al., 1994; Toppino & Gerbier, 2014; Zimmer & Hocevar, 1994). Interestingly, although most studies show significant benefits for spaced practice over massed practice, it should be noted that a few, specifically looking at learned motor skills, show either no effect or a negative effect for distributed practice (García et al., 2008; Panchuk et al., 2013). Despite mixed data on learned motor skills and spaced practice, there exists strong and long-standing evidence that retention of learned, written information is generally enhanced by spaced practice compared to massed practice (Agarwal et al., 2021; Dempster, 1989; Phillips et al., 2019). Perhaps of greater impact from a course planning perspective is the nature of the spaced learning itself: is there an optimal learning format?

Spaced testing versus restudying in medical and health science education

Testing is superior to restudying

The app used in the present study employs both spaced learning and spaced testing. Educational research literature consistently describes the benefits of testing as a learning modality compared to reading and repetition alone. Indeed, testing alone, with no prior exposure to learning materials, may be sufficient for significant learning and memory retention of information, although this “Forward Testing Effect” was relatively recently identified and is still being studied (Larsen et al., 2008; Richland et al., 2009; Yang et al., 2021). However, most studies employ a mixture of exposure to information plus concurrent spaced testing versus free exposure to information alone or as massed or spaced learning (Yang et al., 2021). These studies broadly demonstrate the benefits of repeated, spaced testing over studying alone. In a couple of broad-ranging reviews of the spaced testing literature, the significant benefits of retrieval practice were compared to either restudying under controlled laboratory conditions or to a previously established teaching format, such as traditional lecture in the classroom setting (Agarwal et al., 2021; Yang et al., 2021). Improvement in learning through retrieval practice using spaced testing was observed across all levels of education, from K-12 up to undergraduate higher education programs and medical schools. Researchers did not identify benefits from any single, spaced testing format that was consistent across studies and stated in a final recommendation that “we conclude that educators should implement retrieval practice, with less concern about the precise format or timing of retrieval interventions” (Agarwal et al., 2021: PAGE 23), as virtually all studies showed advantages from the intervention. This is reflected by Yang et al., (2021: PAGE 25), who concluded that “test-enhanced learning generalizes to a variety of test formats,” and all levels of education. The process of information retrieval through consistent spaced testing aids long-term retention of information and provides an evidence-based platform that boosts student success across the spectrum of student experience and educational topics.

Medical educators have recently laid significant groundwork for use of spaced testing in clinical courses. Larsen et al. (2009) employed a crossover study in which students were allocated to two groups studying two identical topics; one group took tests on topic 1 then studied a review sheet for topic 2, while the second group studied topic 1 from a review sheet and took tests covering information in topic 2. Repeated testing resulted in significantly higher scores compared to repeated study alone (Larsen et al., 2009). The act of information retrieval during the learning process reinforces the material being learned but in addition, Larsen points out that “because a final test or application task will also involve active retrieval of information, the active retrieval in the initial testing essentially practices the skill that will be needed later” (Larsen et al., 2008: PAGE 961). So not only does spaced testing reinforce the material learned, but it can also reinforce the final testing procedure in many academic settings. Clearly, this may require intentional design of the formative test to match the summative test, but in a higher education setting formats like multiple choice questions or written answers are frequently used for both evaluative practices.

Separately, but of particular relevance to clinical student cohorts such as those in our dental school who must acquire both didactic knowledge and demonstrate clinical application of that knowledge, the Larsen et al. (2009) study found increased retention and pass rates for in-school testing but failed to find significant improvement in clinical practice due to repeated testing. In contrast, a comprehensive meta-analysis of spaced education in medical continuing professional development programs demonstrated the benefits of distributed learning at all levels of application (Larsen et al., 2009; Phillips et al., 2019). Phillips et al. (2019) identified 14 studies including 5 randomized controlled trials that demonstrated significant benefits to practicing clinicians of spaced or distributed learning provided via online platforms; these benefits included an increase in participant long-term knowledge, significant positive changes in clinician behavior and confidence, a significant increase in clinical proficiency and, in two cases, resulted in a positive enhancement of patient outcomes. Although evidence for the benefits of distributed learning to clinical application is considerably scarcer than for the benefits to academic retention and retrieval, it is nonetheless there.

The “sweet spot” for distributed learning may be at the interface of the academic and clinical worlds with distributed learning and spaced testing becoming the mainstay for many students studying for clinical board certification exams. While learning with flashcards is an old tradition, new flashcard software applications with dedicated spaced or times repetition algorithms, such as Anki (https://apps.ankiweb.net/), are in common use for board exam preparation. These platforms are discussed in more detail below, but two studies illustrate the broader use of spaced testing for board examination success. In a study of clinical board exam pass rates, implementation of spaced testing during an otolaryngology course reflected significantly better scores compared to similar courses where no significant difference existed prior to that implementation (Dabiri et al., 2019). Additionally, using spaced quizzing Wallihan et al (2018) demonstrated higher pass rates for the American Board of Pediatrics certifying examination with the use of bi-weekly quizzes. Individual performance in these weekly quizzes, notably, correlated with board examination results (Wallihan et al., 2018). Therefore, evidence supports the use of distributed learning and testing as appropriate preparation for clinical examination purposes—as it does for broader academic purposes – even if the evidence is less clear for clinical practice.

As mentioned above, flashcard-based software applications that employ both distributed learning and distributed testing are commonly used for certification board exam study but they have also been developed to promote successful progression within academic clinical courses, often for specific medical disciplines. Morin et al. (2019) describe the development of such an app, Spaced Radiology, that standardizes pediatric radiology cases for study using flashcard decks and radiology images (Morin et al., 2019). Similarly, Taveira-Gomes et al. (2015) developed flashcards for the study of basic science topics in a medical school setting. In laboratory experiments comparing spaced testing versus spaced study plus spaced testing, they demonstrated significant increases in memory retention for the study plus test group compared to the group with spaced testing alone. The authors note that “the feedback that is thus formed between the quiz and the study task further promotes the spaced repetition of study and self-assessment sessions and can improve student engagement, the main driver of successful learning” (Taveira-Gomes et al., 2015: PAGE 9).

The success of integrating spaced study and spaced testing emphasizes the importance of optimal instructional design that does not rely on one single modality. Interestingly, the flashcard-plus-testing format used by Taveira-Gomes et al. (2015) is very similar in structure to the format employed by both the Quizlet app described in this study and the open-access flashcard app Anki, the use of which has been shown to correlate to improved scores in medical licensing board exams (Lu et al., 2021; Taveira-Gomes et al., 2015). In that report, Lu et al. (2021) also note that use of the flashcard system resulted in a decrease in the perceived need to re-study material, which the authors suggested may be linked to better memory retention over time. Anki flashcards have also been linked to significant improvement in exam scores in an orthopedic surgery course (Lambers & Talia, 2021). Notably, the authors of this study identify their orthopedic surgery module as “difficult, with a great deal of knowledge necessary to be acquired in a finite amount of time” (Lambers & Talia, 2021: PAGE 1). This parallels our rationale for use of a flashcard system in our accelerated dental program and, with demonstrated benefits of spaced study and spaced testing broadly but also in medical and health education settings specifically, supports our use of flashcards for our dental pharmacology review course.

It is worth noting that compared to medical education programs, evidence for the benefits of spaced testing in a dental school setting is sparse. However, dental educators have demonstrated the value of repeated testing over repeated training on the suturing hand skills of undergraduate dental students. Notably, in four other tested skills students in both control and intervention groups scored the same on average (Sennhenn-Kirchner et al., 2018). While this was a small study group of 36 students in total, the results accord with the mixed results for distributed practice on clinical application compared to academic knowledge retention and retrieval. It may be that the experimental testing method for the study of clinical application leads to greater variation in student evaluation if a calibrated judgment of ability is the end-point, whereas for testing academic knowledge retrieval absolute answers – for example, multiple-choice question responses – provide the significant evidence. In light of these observations, it is reasonable to consider the nature of the testing format.

Does testing format matter?

Differences have been demonstrated for active retrieval, such as short answer writing, compared to passive retrieval as exemplified by multiple-choice testing. Butler and Roediger (2007) compared spaced learning with spaced testing using either multiple-choice questions or short answer questions in an art history class. Although spaced study with review materials and multiple-choice testing both resulted in retained information, for longer-term retention over one month the students using active retrieval achieved significantly higher scores (Butler & Roediger, 2007). It could be argued by Larsen et al.’s (2009) reasoning that the nature of the 1-month capstone test being a short answer gave an advantage to those students who practiced using the spaced short answer tests. Indeed, in a recent systematic meta-analysis, Yang et al (2021) observed that not only does any level of spaced testing—active or passive—provide significantly better learning outcomes than no testing but some evidence suggests that matching spaced testing format and exam format – e.g., multiple-choice questions in both cases – results in increased learning compared to mismatched test and exam – e.g., multiple-choice then short answer (Yang et al., 2021). Given that a flashcard system may well address the issues of spaced learning and active retrieval, it is worth considering the options currently available to both instructors creating or revising courses and to their students.

Software platforms for flashcards and spaced learning

Based on the education literature, we decided to use the Quizlet app in the current study. We chose Quizlet due to its dedicated function for spaced learning and its class/teacher interface however, both spaced learning and testing have been implemented by a variety of software applications. These applications span the range from general enterprise software with associated spaced learning functions to software dedicated to and designed specifically for, the application of spaced learning theory. The following section examines many of the commonly used computer-based options for spaced learning in general, and flashcard-focused learning specifically and we discuss our choice of spaced learning software for application in the present study.

General platforms for quizzing and flashcards

Most current academic institutions use learning management systems (LMS) for the provision of course materials. Systems such as Canvas, Blackboard, Sakai, Moodle, and Google Classroom are well-established LMS options, some with paid access and some with open access (Fathema et al., 2015; Kabudi et al., 2021; Wallihan et al., 2018; Xin et al., 2021). Most of these have either quizzing or assignment tools and most allow integration with third-party applications for active learning. Although these LMS may be used for spaced learning, this is not their focus and the base programs lack the guided retrieval tools of other systems more committed to the spaced learning process. Several studies published on spaced testing and learning in medicine report using QStream, a mobile learning management application dedicated to spaced learning and testing, however, this is an enterprise software and not targeted to individual teachers (Janssen et al., 2016; Robinson et al., 2017). Most of these LMS have been used in the literature for both laboratory-based and classroom-based educational studies and can be used for the study of spaced learning, although some apps have been developed specifically with spaced learning and testing in mind.

iDoRecall and Flashcards + Chegg are two flashcard apps that allow the upload of learning materials by either teacher or student, who can then create flashcards directly from that material, contingent on the copyright or institutional license. Flashcards + Chegg, Cram.com, Brainscape, Anki, and Quizlet all allow sharing of privately created flashcard sets between users. Many of these sets are specific to professional development programs or accreditation board examinations, though they are unofficial study resources with little guarantee of content relevance. For practical pedagogy, it is worth noting that Brainscape and Quizlet allow educators to create classes within the apps and to create their own content that can be shared solely with the class; both also provide educator “consoles” by which individual student progress may be monitored.

The benefits of distributed learning using flashcards and quizzing are widely discussed in medical student discussion forums as they are well established in clinical and health science student communities, where large amounts of information need to be digested in a short amount of time. Flashcard systems specific to medical and health science education are popularly available from Firecracker (Wolters Kluwer) and various medical licensing flashcard applications including Kaplan USMLE (Kaplan Inc), USMLE Mastery (Higher Learning Technologies), Memorang (Memorang Inc), and UWorld USMLE Qbanks (USMLEWorld, LLC). Given our current use of flashcards in a dental school, flashcard and quiz packages are also available for dental licensing exams from Dental Decks (Oakstone Publishing), Kaplan Inc, and Higher Learning Technologies. These all provide copyrighted topic content for the flashcards and are not designed for the addition of content by instructors.

Brosencephalon, a moderated flashcard set based on the Anki software platform, is a very well-established community of medical students who curate and generate flashcards using the Anki app. Created in 2013 by Amreet Sidhu for his own preclinical then clinical medical school studies, this open access deck is widely used by medical students around the world. Many other flashcards sets are available via the Anki app but two created subsequently to Brosencephalon predominate, Lightyear and Zanki. Like Brosencephalon, both are student-curated sets of several thousand flashcards based on the Anki platform and directed toward medical school study. Anki itself is a highly flexible flashcard online application with a powerful distributed learning algorithm; it allows sharing of sets between students as described above for Brosencephalon, Lightyear, and Zanki, but it does not have an intuitive interface. With these considerations in mind, Quizlet was selected for the current study for reasons discussed below.

Quizlet

When looking at software options for flashcards in our current study – a class of international students with a very crowded curriculum – we were cognizant of cognitive load as it applies to the presentation of information. We discuss cognitive load concerning international students below but in short, we aimed to minimize the learning curve and impact of the presentation style for our course. As there are no clear studies comparing spaced learning algorithms for flashcard applications as a decision point, the presentation of an intuitive interface took precedence. Quizlet has a clear, user-friendly interface developed for use by students from junior school through to university.

Like many of the alternatives, it allows the creation and sharing of flashcards and employs spaced learning. Beyond a simple flashcard interface, Quizlet also offers retrieval options of increasing degrees of difficulty in line with the distributed learning practice of guidance fading (Paas & Merriënboer, 2020). In practice, this means that students are initially presented with multiple-choice questions and progress to typed, short answers with less prompting as they demonstrate proficiency. Quizlet also uses variable format tests that can be taken at any time from material learned in the flashcards so students can self-test before any summative tests. Additionally, it employs games in the learning process, specifically matching games of varying difficulty. While gamification is still an area of learning with mixed reports of performance regarding retention and retrieval, it provides an interesting engagement aspect to topic study as students can compete against each other’s timed scores.

Study background and goals

The hallmark of our international dental student program is the front-loaded, fast-paced, and information-dense nature of the didactic teaching in the first two academic terms allowing students to start their clinical experience as soon as possible. International student cohorts are now a familiar feature of many institutions of higher education, bringing these institutions more revenue than domestic students. A broad range of concerns have been identified that potentially share commonality with domestic students but differ by degree and nature of impact, including academic burden, financial anxiety, and social care (Alfattal, 2016; Schlepper, 2004; Ye & Juni, 2018). This is true in both English-speaking countries, as in the current study, and for international students in non-English-speaking countries (Calikoglu, 2018; Johnson et al., 2018; Popov et al., 2012; Ye & Juni, 2018). However, there is evidence that international, foreign language students as a heterogeneous group encounter common challenges beyond those of domestic students and that those challenges are not always effectively addressed by educational institutions (Alfattal, 2016; Johnson et al., 2018).

As might be expected, language barriers, even for international students with deep knowledge of the national language, add to the burden of learning and are a significant stressor and may increase the cognitive load of academic tasks (Popov et al., 2012; Schlepper, 2004). Where language and presentation impact teaching, faculty are often aware of the problem but may lack the specific training or resources to deal with it and an institutional strategy for support of international students is required (Attrill et al., 2016). Nevertheless, at the level of teacher-student instruction, simple ameliorative steps can be taken, like reducing speed and increasing clarity of verbal communication (Schlepper, 2004) and reducing the cognitive load (Attrill et al., 2016).

To address issues of cognitive load in student learning, it is important to recognize that it carries both desirable and detrimental components. Cognitive load is commonly divided into intrinsic cognitive load, which is ascribed to the difficulty of information to be learned; extraneous or extrinsic cognitive load, which relates to the information delivery mechanism; and germane cognitive load, which relates to the act of learning and integration with existing knowledge (de Jong, 2009). Chandler and Sweller (1991: PAGE 295), in an early application of cognitive load theory to instructional design, suggested that “presentation techniques frequently result in high levels of extraneous cognitive load that influence the degree to which learning can be facilitated. It follows that information should be presented in ways that do not impose a heavy extraneous cognitive load,” (Chandler & Sweller, 1991). They demonstrated from experimental data that redundant instructional information may not simply add nothing to the capacity to acquire the intended schema but may be actively detrimental to that learning (Chandler & Sweller, 1991; Sweller, 1988). de Jong (2009: PAGE 109) takes that further, arguing that familiarity with the instructional framework effectively reduces the extraneous cognitive load suggesting that “a learner’s awareness of specific conventions governing the construction of learning material assists with processing and thus reduces extraneous cognitive load,” (de Jong, 2009).

Whether paring back extraneous instructional material or using an instructional format that the student is familiar with, there is clear evidence that student capacity for the learning and retention of schema is increased where they need to focus less on the process of instruction and more on the taught content. As noted above, extraneous cognitive load encountered by international students studying in a language other than their native tongue poses unique challenges for both the students and their instructors. Tailoring instructional design based on cognitive load theory to address these challenges for non-native speaking students was a driving force for establishing the conditions for the present study and future development of the international student dental pharmacology course described in this study.

Methods

Setting

This study is designed to determine student perceptions of an iterative learning app introduced to a Clinical Pharmacology and Pathology review course. The University of the Pacific Arthur A. Dugoni School of Dentistry offers a two-year Doctor of Dental Surgery (DDS) program for internationally-qualified dentists wishing to practice in the United States of America, with an annual intake of 26 students. The preclinical portion of the course is highly accelerated to allow students into the clinic after only six months. Structuring critical information over that period is a challenge for both the instructors to deliver and the students to acquire.

In the present study, we describe the use of Quizlet, a commercially-available, iterative learning and self-testing platform in the pharmacology section of the above course, and present the results of a survey of student attitudes towards its use. In our study, Quizlet was used to present review material to an international post-graduate student cohort in a pharmacology review course within an accelerated US dental school program.

The specific goals of the study were to.

Determine international student attitudes towards the use of Quizlet in an advanced higher education setting

Determine how those students are using the app

Assess their perception of information content level within the flashcard sets.

Intervention

The ADA Dental Drug Handbook (American Dental Association) was used as source material. The chapters deal with separate classes of drugs and from each chapter the instructor created a discrete flashcard set relating to that drug class. Sets were released every two weeks, generally one or two sets at a time such that students were required to learn between 14–38 flashcard terms and definitions over each two-week period. The course duration was two quarters, 6 months. Class progress through the sets was monitored using the Quizlet app itself. Bi-weekly office hours dedicated to the course allowed students to bring any questions relating to the course material to the instructor.

The ability to monitor student progress through the course was necessary to ensure compliance with course requirements and the Quizlet app, uniquely amongst the available flashcard software, provided that function. Quizlet has both free access and premium access options, the latter option including “Teacher controls” that allow the creation of discrete classes and the monitoring of student progress within those classes. The app incorporates flashcard settings and activities that repeat the flashcard information in different formats and at different levels of active learning. It also provides two games that incorporate flashcard material. Match is a relatively simple pairing game in which the student moves a definition across the screen to its appropriate term under timed conditions while the more challenging Gravity requires the student to type the definition into a text box before the term disappears as it falls from the top of the app window and off the bottom. A test function that uses all the Quizlet learning formats, other than the games, provided students with a tool for self-assessing overall progress. Notification of progress stage in each learning Quizlet tool and grading of success in the self-test was made instantly available to the student.

Although all Quizlet functions were recommended for use, only the self-test function was an absolute requirement The app replaced an online self-study module comprising a large volume of presentation slides with no additional commentary. The pharmacology aspect of the course was designed as both a review of known drug information and to familiarize students with clinical drug details, approved in the USA, that might differ from those in their countries of origin.

Participants

University of the Pacific Institutional Review Board approved this study under Approval Protocol IRB2021-85; a waiver was granted as the study did not meet the designation of human subject research.

Students were eligible to participate in this study if they had completed the international student pharmacology program at the University of the Pacific Dugoni School of Dentistry over the last two academic years. Participants were recruited from two consecutive classes of 25 and 26 students, respectively, where Quizlet was employed as the primary pedagogical method as described above. Demographic details are presented in Table 1. Over half of the students recorded India as their country of birth (56.8%), with the remainder of the cohort composed of 1–3 students from 16 other countries. Women comprised 80% (41) of the cohort, compared to 10 male students (20%). The date of initial degree award and number of degrees was broadly indicative of a mature cohort of students, although ages were not available or surveyed; initial dental degrees were awarded as early as 2003 up until 2018. All participants reported a primary dental degree and 43% reported a second higher degree, two students reported a third degree. Secondary and tertiary degrees were either Master's or Ph.D. level.

Table 1

Class demographic information, obtained from school administrative records

		Class 1	Class 2	Combined
Country of birth	China	1	1	2
	Egypt	1	0	1
	India	14	15	29
	Iraq	1	1	2
	Lebanon	0	1	1
	Moldova	1	0	1
	Nepal	1	0	1
	Nigeria	0	1	1
	Pakistan	1	2	3
	Russia	1	0	1
	Saudi Arabia	0	2	2
	Serbia	0	1	1
	Slovenia	1	0	1
	Sri Lanka	0	1	1
	Syria	1	0	1
	Unites States	2	1	3
	Yemen	1	0	1
Gender	Female	23	18	41
	Male	3	7	10
Degrees held	Primary dental degree	26	25	51
	Secondary degree	13	9	22
	Tertiary degree	2	0	2
Year of Dental Degree		2003–2016	2006–2018

Questionnaire and data collection

Two academic years of students having completed the course under study were sent a pre-contact email to explain the purpose of the survey and to assure anonymity. The survey was created using Qualtrics software, which was also used to create an HTML link. Eligible students were sent an email with the link to the survey questionnaire regarding their perceptions of the Quizlet app itself and the design and use of the pharmacology content presented using the Quizlet app. The questionnaire (Supplemental Table 1) was primarily compiled and adapted from Lewis J.R. “IBM Computer Usability Satisfaction Questionnaire” (Lewis, 2021), and (Lund, 2001), “Measuring Usability with the USE Questionnaire” for assessing app usability and (Rodis et al., 2016) for assessing perceived efficacy of material presentation. The Qualtrics survey software compiled responses and was used to collate the survey data.

Results

Response numbers

98% of the 51 students making up the study cohort responded to the survey. Four responses were removed where duplicates were found to be from the same IP address, leaving 47 recorded responses. One respondent refused consent, so a final tally of 46 anonymous respondents from two consecutive class years was analyzed in this study.

Computer skills and access mode

Students were asked to assess their computer skills. Their self-reported responses were broad, ranging from 30/100 to 100/100, with 100 indicating the highest computer skills (Table 2). The mean value was 77/100 and the mode was 90/100, suggesting that most students were equipped with appropriate experience for using computer access to the study material. Respondents were also asked about their mode of accessing Quizlet, as the program has a dedicated computer interface accessed from a browser and a separate one for the Quizlet mobile app. Over half the students accessed Quizlet using a laptop (57.8%) with a mobile phone being the other major access method (32.8%), the remaining 9.4% of students used a desktop or tablet (Fig. 1).

Table 2

Self-reported computer proficiency on a continuous scale of 0 (lowest) – 100 (highest)

Min	30
Max	100
Mean	77
Std Dev	17.27
Variance	298.22
Response Count	46

Fig. 1

Device usage: Students were asked “What device do you use for accessing the Quizlet app?”, in total 64 responses were recorded indicating the use of multiple devices by individual students

Attitudes to Quizlet

Nearly two-thirds of respondents had little or no prior experience with Quizlet, with 59% somewhat/strongly disagreeing with the statement “I have previous experience of the Quizlet app prior to the Pharmacology course” (Fig. 2). 84% of respondents somewhat/strongly agreed that the Quizlet interface is easy to navigate. Notably, over 86% of respondents agreed with the statement that “Overall, Quizlet was useful for their learning” (Fig. 2).

Fig. 2

Students were asked to indicated level of agreement for the following statements: “I have previous experience of the Quizlet app prior to the Pharmacology course” (RED), “The Quizlet website interface is easy to navigate” (ORANGE), and “Overall, Quizlet is useful for my learning” (GOLD), for each question, 44 responses were recorded

We asked, more broadly, whether students would “recommend the Quizlet app for other courses.” Over 90% of students somewhat/strongly agreed with this statement. Interestingly, of the four students who disagreed with this statement, the only clear relationship with other unique response survey parameters was that all four of these students indicated strong disagreement with having prior experience with Quizlet (Fig. 2).

Quizlet provides seven different tools to aid learning including flashcards (Flashcards), several with different written or multiple-choice response elements (Learn; Write; Spell), a self-testing tool (Test), and two tools presenting the information in a game format (Match; Gravity), both using forms of information matching. We surveyed students regarding perceived value of each of these tools on a 0–100 scale (Table 3). Flashcards, Test, and Match all received mean scores of over 80/100 with scores ranges of 19–100 for these tools. The Learn tool was scored at 78/100 but all others received ratings below 66/100. Notably, the only tool to receive a zero rating from any respondent was Gravity with two students rating its utility as 0/100 although its score ranged from 0 to 100.

Table 3

Perceived value of each Quizlet learning tool. Students were asked to rate their perceived value for each tool on a continuous scale of 0 (lowest) – 100 (highest)

Field	Minimum	Maximum	Mean	Std Deviation	Variance	Count
Flash cards	20.00	100.00	83.95	20.98	440.23	44
Learn	2.00	100.00	78.00	24.50	600.42	43
Write	5.00	100.00	65.39	26.76	716.04	41
Spell	5.00	100.00	59.52	27.54	758.34	42
Test	19.00	100.00	88.31	18.12	328.21	42
Match	20.00	100.00	82.56	20.59	423.97	43
Gravity	0.00	100.00	65.88	32.14	1032.66	40

Attitudes to the Quizlet format of Pharmacology material

We asked specific questions relating to the overall value of the pharmacology content in the flashcard sets and the level of information on the individual cards. Table 4 and Table 5 show the questions and breakdown of responses for the respective sections. Regarding the overall value of the Quizlet pharmacology sets, there was a greater than 88% agreement that the sets both provided a good review of clinical dental drug knowledge and that they also provided new clinical dental drug knowledge (Table 4). Thus, Quizlet was seen to be useful for enhancing existing knowledge as well as providing new information. There was also consensus that the individual cards contained sufficient information (82%), that there was an appropriate level of information (86%), and that there was a sufficient number of cards per set (79%) (see Table 5).

Table 4

Student level of agreement with statements regarding the overall value of the Quizlet Pharmacology Review sets. Presented as percentage of respondents (%) and number of responses (#)

Question	Strongly disagree	#	Somewhat disagree	#	Neither agree nor disagree	#	Somewhat agree	#	Strongly agree	#	Total
The Quizlet sets provided a good review of my clinical dental drug knowledge	2.33%	1	2.33%	1	6.98%	3	34.88%	15	53.49%	23	43
The Quizlet sets provided me with new clinical dental drug knowledge	2.33%	1	2.33%	1	6.98%	3	41.86%	18	46.51%	20	43
My dental drug knowledge was improved after studying the Quizlet sets	2.33%	1	4.65%	2	6.98%	3	41.86%	18	44.19%	19	43
The sets enhanced my knowledge of dental drug interactions and adverse reactions	4.65%	2	2.33%	1	9.30%	4	25.58%	11	58.14%	25	43
The sets enhanced my knowledge of dental drug dosing	2.33%	1	4.65%	2	6.98%	3	25.58%	11	60.47%	26	43
Overall the information provided in the Quizlet sets was useful for clinical practice	2.44%	1	2.44%	1	9.76%	4	29.27%	12	56.10%	23	41

Table 5

Student level of agreement with statements regarding the information content of the Quizlet Pharmacology Review sets. Presented as percentage of respondents (%) and number of responses (#)

Question	Strongly disagree	#	Somewhat disagree	#	Neither agree nor disagree	#	Somewhat agree	#	Strongly agree	#	Total
The information content level in each topic set was appropriate	2.33%	1	9.30%	4	4.65%	2	37.21%	16	46.51%	20	43
There were sufficient number of cards in each set	4.76%	2	4.76%	2	11.90%	5	35.71%	15	42.86%	18	42
The information content level in each card was appropriate	4.76%	2	2.38%	1	7.14%	3	38.10%	16	47.62%	20	42
There was sufficient information on each card	2.56%	1	5.13%	2	10.26%	4	38.46%	15	43.59%	17	39

Open comments

We asked the summative question “What did you like most about the Pharmacology Quizlet sets?” Representative comments referred to “easy” and “ease” as shown in the word-frequency summary of comments (Table. 6a). One respondent noted that “it's a good resource for us to repeat again and again, and quantitively measure our memorization of the facts,” which highlights the formative aspect of Quizlet allowing progress to be self-monitored. The ability to study whenever convenient was noted in several comments “makes it easy to learn on the go” and “I can do it when commuting.”

Table 6

Word frequency from student written responses greater where word was repeated in more than two responses. Responses to A. “What do you like most about the Pharmacology Quizlet sets?” and B. “What suggestions do you have to improve the Pharmacology Quizlet sets?” Other than the word “not” all articles, prepositions and connectives were excluded

A
easy	8
ease	5
review	5
again	4
flashcards	4
access	3
fun	3
quick	3
remember	3
B
more	9
none	4
content	3
information	3
not	3

We asked a further summative question, “what suggestions do you have to improve the Pharmacology Quizlet sets?” The word-frequency summary of comments highlights “none” as the most common response to this question (Table. 6b); because the option to leave the response blank was available, yet students actively indicated “none” when asked if improvement was needed, we interpret this as meaning “no suggestion for improvement,” thus validating the flashcard sets. Of those who indicated a need for improvement, the focus was on the level of content and not on the function of the Quizlet app. Comments either requested more flashcards per set or more information per flashcard; “add some more detailed info” and “more extensive information” were representative comments. Of the 19 responses to this question, only three referred to the function of the Quizlet app. One comment was “the test takes into consideration the exact spelling and punctuations (sic) which doesn’t make any sense,” referring to the fact that Quizlet tools do not recognize variations or mistakes in spelling, perhaps unsurprising as the pharmacology flashcards were necessarily complex, reflecting the nature of the subject. Thus, a student may understand a piece of information at the conceptual level but where Quizlet requires a written input an inability to complete a test question verbatim results in a negative score that may not reflect understanding. The other comments referring to the Quizlet app itself were “overall the app is beneficial for review, not for the learning tool (sic)” and “better integration of the testing tool as the option are repetive (sic) and not truly indicative of your knowledge.”

Results summary in relation to study goals

Goal 1. Students were overwhelmingly positive about the use of the Quizlet app in the pharmacology review course. All survey measures relating to the Quizlet app itself were balanced heavily in favor of each tested aspect of the app. Students found the interface intuitive and easy to use irrespective of computer skill or prior experience with the Quizlet app. They also broadly found the learning tools within Quizlet to be useful to their learning. This finding is likely transferrable between similar student cohorts in different courses.

Goal 2. Primarily students are using laptops and mobile phones to access Quizlet. One aspect of Quizlet that students found particularly favorable was its ease of access to learning materials. Notably, both laptop and mobile phone use predominated over other access routes for Quizlet. Quizlet provides both a computer-based interface for laptops and a mobile phone optimized interface which likely accounts in part for this perceived ease of use. As laptop and mobile phone use are ubiquitous it is likely that this observation is transferrable between student cohorts in different courses.

Goal 3. Students generally perceived their experience of the app as easy referring to both the ease of use app and the clarity of the content material. Some students would like more content in the flashcard sets. The imprecation for more course material in this flashcard format suggests that we overestimated the cognitive load effected by the flashcards and that we can expand the content of the flashcard decks. This observation is likely transferrable to a specific set of students in a course like ours that reviews material rather than provides entirely new material.

Discussion

The principles of active learning and spaced retrieval have been understood for a long time, but availability of e-learning adapted software and formal adoption of such software by institutions of higher education is relatively recent. Over the last two years, the Covid-19 pandemic accelerated interest in computer and cloud-based software to aid asynchronous and distance learning (Hira & Anderson, 2021; Pollom et al., 2020; Wang et al., 2021). Despite extensive positive evidence from the fields of neuroscience, psychology, and education research, however, there remains mixed acceptance from students in higher education towards e-learning broadly and spaced learning software specifically (Wang et al., 2021; Lu et al., 2021; Veremis et al., 2021; Muhammad Sajid Mehmood et al., 2021). Students in higher education who use active learning and spaced retrieval to study tend to do better or have greater user satisfaction than those who do not (Deng et al., 2015; Lu et al., 2021; Tsai et al., 2021). In a study of dental students using the Anki flashcard system, Veremis et al (2021) suggest that students who do not use it may simply fail to understand the value of active learning and spaced retrieval in memory retention (Veremis et al., 2021). Therefore, in implementing Quizlet, which offers tools using different levels of active learning and spaced retrieval, we wished to assess student satisfaction with that software through their perception of the value of the app and its tools and the facility of its use. With this data, we aim to improve our application of the software within the course.

In our study, students recorded high approval of the Quizlet flashcard system. Although the average self-scored technology level was relatively high at 77/100, around 66% of students had little or no experience with Quizlet. Given that the course and the use of an unfamiliar educational technology were made mandatory, it might have been reasonable to expect more resistance to the novel course format than we encountered. These results may be explained to some degree by the Technology Acceptance Model (TAM) originally described by Fred Davis (Davis, 1989) in which “perceived usefulness” and “perceived ease of use” were identified as indicators of acceptance of new technologies. Many other models have arisen from the fields of sociology, social psychology, and psychology and indeed this model has been adapted and extended since its initial description however the original basis for the model has remained remarkably consistent under different test conditions (Salloum et al., 2019; Gunasinghe et al., 2019).

The relative simplicity of the TAM and even some of its extended variants is attractive in this instance as the use of Quizlet was not a matter of student choice. Thus, the practical ease of use and perception of usefulness may be expected to play a larger role in student acceptance of the software than perhaps broader social and psychological factors employed in other acceptance models. As an extension of the TAM, “perceived value” has been included with “perceived usefulness” and “perceived ease of use” as an explanatory variable in several studies (El-Wajeeh et al., 2014; Hazen et al., 2015). Perceived value was examined in our survey with 86% and 90% of students respectively saying that Quizlet was useful to their learning and that they would recommend it for other courses.

At the level of individual tools within Quizlet, it was notable that the more difficult the learning tool—i.e., the lower its ease of use—the lower the mean perceived value of that tool. On a scale of 0—100 the mean perceived value for the passive learning Flashcard tool (83.95) was conspicuously higher than that of the active learning tools, Learn (78), Write (65.39), and Spell (59.52). The differences were not statistically significant, and indeed the study was not powered for comparative statistical analysis, but the trend was notable. This was also reflected in the assessment of the two gaming tools, the relatively easy Match, and the more challenging Gravity. The perceived value mean score for Match was 82.56 while that of Gravity was 65.88. Mac Callum et al. (2014) equate perceived ease of use for new technology with the freedom from effort related to that technology (Mac Callum et al., 2014). This may explain the apparent trends in perceived value related to passive (easier) versus active (harder) tools within Quizlet. Higher sample n-values would help discern the validity of this trend however the course admission limits that parameter with an acceptance of around 25 students per year, we will continue collecting longitudinal data.

Survey responses to a question asking for comments on course improvements also imply acceptance of the Quizlet app itself. Where comments for improvement were made, most requested more topic information in the card sets, and not for a different presentation mode, suggesting that students found the flashcard system not only useful but worth expanding. Only one out of nineteen comments referred to the function of the application, although it did highlight a real limitation of Quizlet flashcards. When a student must respond with a written answer only an exact spelling is accepted as correct. While correct spelling is important for essential terms or definitions, the misapplication of a word such as “a” instead of “the” would also generate a response indicating a wrong answer. There are steps available to mitigate this issue, a student may override the “incorrect answer” response. Additionally, the instructor can define parts of the information on any card to be accepted as correct instead of the entire term or definition.

As Quizlet, like virtually all the spaced learning apps, is used as a formative learning tool it should be made clear to students that issues like sentence parsing problems are secondary to accurate retention of topic-specific information through the process of spaced recall. To this end, it may be argued that how students are introduced to spaced learning may be as important as the learning process itself as students may not appreciate the value of the process and instead focus on minutiae (Veremis et al., 2021). Indeed, broader communication both about the process but also the content is required when implementing distance learning tools such as Quizlet. Examining student experience of remote instruction in higher education during Covid-19, Ives (2021) identifies instructor communication and student engagement as parameters weakened by transition to an online format from traditional in-person lecture (Ives, 2021). Communication of academic expectations is important whether face to face or online. Less obvious to many instructors may be the need for clear communication regarding expectations for the use of the new online educational technology itself, whether it be directed to online teaching generally as explored by Ives, or internet-based educational technologies such as Quizlet (Ives, 2021; Veremis et al., 2021).

Our data on mode of use was also revealing. Approximately two-thirds of the class employed laptops to complete the study sets and approximately one-third used mobile phones. We did not differentiate these results and there is certainly overlap in these two categories where students used both, as there were 64 total responses to this question and only 46 students responding. The immersion of a student in an online task is one predictor of their acceptance of the technology and this is described by the concept of flow (Liu et al., 2005). Flow, in its simplest form, is the state of focus and immersion of an individual in a given task to the exclusion of all else (Csikszentmihalyi, 1990). Liu et al (2005) demonstrated that “e-learning materials presentation types are related to perceived usefulness of the technology” when they compared information presented as combinations of text, audio, and video. They showed that concentration on the task along with the perception of usefulness both positively correlate with an intention to use the technology (Liu et al., 2005). Thus, in our study, the ability of students to study the material either on the go or in one place using a phone or laptop potentially enhanced the flow experience compared to the more limiting extended slide presentation that comprised the prior course construction. Moreover, the slide presentations were purely text-based while the Quizlet presented material via text, audio, and visual interaction, if not video, and so in agreement with Liu et al. (2005) this mixed presentation format may account for some of the acceptance related to flow.

The acceptance of Quizlet as a teaching medium by our international student cohort seems to be driven by the ease of use of the flashcards. Nearly 50% of the comments in response to the question “what do you like most about the Pharmacology Quizlet sets?” referred specifically to ease of use, and directly included the word “ease” or “easy.” Our primary aim was to reduce cognitive load within the course as this was seen as a major impediment to learning and understanding before we implemented the flashcard format. In simplifying the format, we did not also simplify the content, but we did reduce the content volume to accommodate the flashcard model. The survey returns have provided a goal for future development: maintaining the level of content complexity but increasing the volume of content. Although there is research on the optimal timing of distributed practice and spaced retrieval, there is no guidance around optimal distributed practice using flashcards in higher education specifically (Toppino & Gerbier, 2014). To this end, the increase in card numbers will be empirical and follow-up surveys are planned to assess the effects of this change in future student cohorts.

Conclusion

The current study demonstrates that Quizlet is an appropriate educational technology for course development in a professional healthcare tertiary education setting. Furthermore, the international student cohort recruited for the study found the application interface easy to use and the included study material easy to learn as a consequence. As students attending this course were qualified dentists, either recent graduates or practicing clinicians, all of whom had taken the American Dental Association board exams which included pharmacology, the pharmacology course was designed as an online, self-study review course for international students to review essential dental drug information specific to clinical application in the United States of America. In practical terms this allowed us to focus on discrete facts without requiring large amounts of background subject instruction. For this purpose, we felt that a flashcard app such as Quizlet would be a good medium with which to provide that information in a format that maintains a relatively low cognitive load. Our use of Quizlet is supported by the consistently positive responses received from two sequential cohorts of students surveyed after taking this course. Although specific to a small cohort of students and for a discrete purpose our study provides support for use of the Quizlet app in a higher education setting for international students at a professional healthcare school. Future work should aim to assess the benefits of instructor applied Quizlet and other flashcard software applications on long term knowledge retention for international students in tertiary health settings.

Below is the link to the electronic supplementary material.

Supplementary file1 (PDF 81 KB)