Motivating Students to Engage in Preparation for Flipped Classrooms by Using Embedded Quizzes in Pre-class Videos.

BACKGROUND: The success of flipped classrooms is dependent upon students' preparation prior to class, the lack of which is the most common challenge associated with this teaching methodology. To mitigate this limitation, it is important to develop and assess methods of engaging learners during pre-class activities.
OBJECTIVE: To determine if quiz delivery method (embedded throughout versus after pre-class videos) affects students' knowledge retention, grades, and video viewing behavior.
METHODS: Participating students were randomized to take quizzes, either during pre-class videos via Panopto™ (EQV) or after pre-class videos in the traditional manner via Moodle™ (TMM). Outcomes assessed included students' knowledge retention, scores on pre- and post-class quizzes, and pre-class video viewing behavior (total views and minutes viewed per student) during a three-week period. Having experienced both quizzing modalities during the semester, the perceptions from students in the EQV group were surveyed.
RESULTS: Baseline assessment results of both groups (n = 27 per group) were comparable with a median score of 33% (IQR: 17, 50) in both groups. Performance was also similar on knowledge retention [TMM: 67% (50, 83) vs. EQV: 83% (50, 83)], pre-class quiz scores [TMM: 90% (87, 97) vs. EQV: 93% (90, 95)], and post-class quiz scores [TMM: 93% (80, 100) vs. EQV: 87% (80, 100)], while students in the EQV group had more total views [10 (8, 12)] vs. [5 (2, 11)] and minutes viewed [71 (36, 108) vs. 35 (15, 81)]. Results from the perception survey administered to students in the EQV group (74.1% response rate) indicated a preference for embedded quizzes overall (58%) and for class preparation (75%) when compared with post-video quizzes.
CONCLUSION: Students' knowledge retention and performance were similar in both EQV and TMM groups, though students in the EQV group were more engaged with videos and most of them preferred this quiz delivery. Using embedded quizzes for formative rather than summative assessment might be an appropriate mechanism to encourage students' viewing of pre-class videos and their preparation for flipped classes. © Individual authors.

INTRODUCTION

Utilization of flipped classrooms, in which students review content prior to an active-learning class session, has increased in pharmacy education over the past decade.[1-9] While methods for presenting pre-class information to students are myriad,[1-11] there are many documented benefits of flipped classrooms.[2,10,11] One clear lesson is that the success of the flipped classroom is dependent upon students’ prior preparation so that they can engage during class.[9,11-13] The most commonly reported challenge educators encounter when using flipped classrooms is lack of students’ preparation.[11]

Based on the literature that videos are the most frequently utilized method of presenting pre-class content,[11] educators appear to believe this method will encourage student participation. When coupled with quizzes, video viewing time, and thus engagement with pre-class materials, has been shown to increase.[14] Video-recording technology has advanced to the point that software now allows the incorporation of quizzes directly into recorded lectures.[15] Given that students benefit from reviewing pre-class materials prior to a flipped classroom session,[9,11-13] it seems likely that encouraging students to watch pre-class videos by embedding required quiz questions would improve their engagement and retention and also possibly their enjoyment. However, this hypothesis must be assessed to allow for appropriate utilization of the combination of teaching and assessment methodology.

Although there is existing literature that suggests embedding quiz questions in pre-class videos can increase student engagement and enhance student opinion,[16-18] it is unknown whether embedding quizzes within pre-class videos increases student video use and thus learning, a necessary outcome in educational research.[19] Therefore, the purpose of this study was to conduct a randomized controlled trial to determine whether embedded quizzes in pre-class videos can improve students’ performance, retention, and engagement in a flipped pharmacy course.

Directly comparing videos using embedded quizzes with videos not using embedded quizzes will help establish the educational value of this mechanism by determining the effect on student learning.

METHODS

As part of the required curriculum, first-year pharmacy students were enrolled in the one-credit hour Introduction to Drug Information course focused on teaching students the systematic approach to drug information (DI) and resources available for answering DI questions. Course content, primarily developed by the course coordinator and medical librarian, was taught via a flipped format, and the structure was automated via the learning management system (LMS) so that new content was available to students on a weekly basis. Guest faculty were utilized during specific weeks to highlight DI use in other pharmacy practice settings via pre-recorded videos. Pre-class content included videos linked directly in the LMS and accompanying handouts. Videos were created utilizing Panopto™ (Panopto: Seattle, Washington) and were typically either recorded PowerPoint® (Microsoft Corporation: Redmond, Washington) slides or screencasts displaying the instructor’s screen while narrated instructions were provided. The number of videos and instructors used each week and the duration of each video during the course of the entire semester is reported in Table 1.

Table 1.

Description of Pre-Class Videos during Fall 2018 Semester

Week	Content	Number of instructors	Total number of pre-class videos	Length of videos (minutes and seconds)
1	Background questions and categorization	3	4	5:06, 7:44, 9:40, 10:36
2	Tertiary resources	2	3	9:14, 9:13, 9:07
3	Online compendia	3	3	11:48, 10:55, 7:11
4	Internet resources	1	3	8:56, 27:11, 23:42
5*	Mobile apps	1	2	8:16, 12:51
6*	Literature searching (basics)	1	4	6:59, 11:06, 25:19, 10:11
7*	Literature searching (advanced)	1	3	12:00, 18:52, 13:12
8	Drug information center question	2	2	2:26, 7:34
9	Community question	2	2	11:02, 9:05
10	Patient case question	2	2	5:07, 8:05
11	Hospital question	2	2	5:08, 8:40
12	Formulary question	2	2	8:54, 12:06
13	Clinical question	2	2	15:12, 8:13
Total (average per week)		24 (1.9)	34 (2.6)	370:41 (10:54)

Study Week

Pre-class content was released to students on Monday morning at 12:00 AM. All students had to complete a 10-question multiple-choice quiz by 11:59 PM on Thursday of the same week prior to attending an in-person lecture. The in-person lecture was taught from 11:00-11:50 AM each Friday morning focused on practical application of pre-class content using hands-on, active learning activities. All students took a five-question multiple-choice quiz after class via the LMS before Sunday at 11:59 PM of the same week as a summative assessment for the week’s content. Pre- and post-class quizzes were graded and contributed to the students’ final grade.

Using this flipped classroom structure, a randomized, comparative study was conducted to evaluate the impact of taking pre-class quizzes through embedding within video (intervention) compared with the traditional manner of taking the assessment via LMS (control). The intervention was designed based on the flipped classroom pedagogical framework and the cognitive information processing (CIP) theory, in which new information stored immediately in short-term memory is transitioned to long-term memory through encoding and rehearsal.[20] By introducing information in pre-class videos and then revisiting that information immediately and repetitively in live lectures, students are able to develop long-term memory of content taught in this course.

At the beginning of the fall 2018 semester, students in the study answered demographic questionnaires, including those assessing factors thought to potentially influence time dedication to pre-class participation, specifically marital, employment status, and previous experience with flipped classrooms. These students then took a 6-point baseline assessment consisting of four multiple-choice questions, one of which required multiple answers. In an attempt to validate the baseline assessment, a small number of advanced learners (e.g., fourth-year students, residents) completed the quiz prior to its use in the study. Students participating in the study did not receive feedback on the baseline assessment, and questions utilized were not asked during the study period. Students who were unwilling to participate in the research study took all pre-class assessments in the traditional manner via the LMS, and data associated with those students were not included in this study. Students that consented to be part of the study were randomized using a random number sequence generator in Excel® (Microsoft Corporation: Redmond, Washington) and placed into either the control or intervention group based on the number assigned.

In order to ensure video-lecture and quiz-question consistency, this comparative study focused on a subset of lectures taught exclusively by the medical librarian in conjunction with the course coordinator during weeks five through seven. Participating students completed pre-class quizzes in the traditional manner via Moodle™ (Moodle Pty Ltd: West Perth WA, Australia) (TMM) or the new method of simultaneously watching content and answering quiz questions directly embedded throughout the pre-class videos (EQV).

For the intervention, students in the EQV group took pre-class quiz questions embedded throughout the videos, not only at the end (TMM). Students were required to view the videos of their own volition regardless of assignment; no mechanism for forced viewing was employed other than the embedded quizzes presented to the EQV group. If students were in the EQV group they needed to watch these videos on a desktop or laptop computer and should not watch them on a mobile device because it would prevent them from answering the interwoven quiz questions due to a video viewing software limitation. This was communicated to students verbally by instructors each week. Furthermore, a disclaimer was placed in text below each video with embedded quizzes. Students in the TMM group were allowed to watch the videos on any device they chose. The pre-class quizzes for both groups were identical except for method of delivery.

After the class, all students took the identical post-class quizzes via the LMS. All quizzes were multiple-choice format, and scores and explanation regarding correct answers, were not provided to either group until after the due date of the quiz to mitigate potential academic dishonesty. At the time of this study, Panopto™ Quizzing and Moodle™ (our LMS) Gradebook were unable to be integrated. This required the course coordinator and librarian to manually transcribe individual student answers from Panopto™ to Moodle™ on Friday mornings before the live lecture so scores and justification for correctness of answers were available to all students simultaneously regardless of group assignment. The medical librarian thematically reviewed answers provided on the pre-class quizzes and spent the first few minutes of the live lecture clarifying trouble spots for students.

The primary outcome of this study was students’ end-of-semester retention, based on the performance on the baseline assessment, which was administered to participating students at the end of the semester. Secondary outcomes included grades on the pre- and post-class quizzes, total videos viewed, and time students spent viewing videos. Additionally, because the EQV group experienced both methods of quiz delivery during the semester, willing students in the EQV group were surveyed for their opinions and engagements for embedded quiz method.

Data collection included basic demographic information, grades on the baseline assessment, pre- and post-class quizzes, student engagement via total views and minutes of pre-class videos viewed over three study weeks per group, and grades on the baseline assessment administered post-course to assess retention. Between-group comparisons of continuous data were conducted via Wilcoxon Ranked Sum tests due to the small sample sizes per group and the skewness of the distribution of outcomes of interest. Student survey data were reported via descriptive statistics. For comparisons of nominal data between groups, Fisher’s Exact Tests were used. Analyses were generated using SAS software, Version 9.4 of the SAS System for Windows (Cary, NC, USA). The Institutional Review Board from Medical University of South Carolina approved this study.

RESULTS

A total of 54 out of 68 students (79%) participated in this research study. Students in both groups had similar scores on the initial baseline assessment [median: 33% (IQR: 17, 50) for both groups]. Demographics and baseline performance per group can be found in Table 2. There were no statistically significant differences between groups with regard to demographic and baseline performance variables.

Table 2.

Demographics and Baseline Performance by Groups

	TMM (n = 27)	EQV (n = 27)	p value
Gender (% female)	85.2	77.8	.73
Marital status (% single)	92.6	88.9	>.99
Employment status (% working)	37.0	55.6	.27
Age (median [IQR])	22 [21. 24]	22 [21, 23]	.80
Previous experience with flipped classroom (n, % yes)	11, 40.7	7, 25.9	.39
Prefer flipped (n, %)	4, 36.4	3, 42.9	.17
Prefer traditional (n, %)	6, 54.5	1, 14.3
No preference (n, %)	1, 9.1	3, 42.9
Anticipated viewing of 76-100% of pre-class videos (%)	74.1	77.8	.86
Baseline assessment score (%, median [IQR])	33 [17, 50]	33 [17, 50]	.82

TMM=traditional Moodle™ method; EQV=embedded quiz videos

IQR=Interquartile Range

Wilcoxon Ranked Sum Tests and Fisher’s Exact Tests were used to determine significance between groups, defined as p<.05

value

Results from the comparison of performance and engagement outcomes are presented in Table 3. Students’ performance on pre- and post-class assessments showed that students performed similarly on the individual quizzes both before (TMM: 90% vs. EQV: 93%) and after (TMM: 90% vs. EQV: 87%) class regardless of quiz presentation.

Table 3.

Students’ Performance and Video Viewing Behavior over Three Study Weeks by Group

Outcome (Median [IQR])	TMM (n = 27)	EQV (n = 27)	p value
Aggregate pre-class quiz grade %	90 [87, 97]	93 [90, 95]*	.57
Aggregate post-class quiz grade %	93 [80, 100]	87 [80, 100]	.62
Total views of nine pre-class videos per student	5 [2, 11]	10 [8, 12]	.03+
Total minutes viewed of nine pre-class videos per student†	35 [15, 81]	71 [36, 108]	.09
Aggregate post-course retention assessment (%, median [IQR])	n = 25	n = 19	.91
	67 [50, 83]	83 [50, 83]

IQR=Interquartile Range; TMM=traditional Moodle™ method; EQV=embedded quiz videos

n = 26; 1 student did not complete any of the 3 pre-class quizzes

Total number of minutes for 9 pre-class videos = 118:46

Wilcoxon Ranked Sum Test was used to determine significance between groups, defined as p<.05

Students in the EQV group were significantly more likely to view pre-class videos than those in the TMM group (10 vs. 5, P = .03) and, while not statistically significant, watched more total minutes of the videos (71 vs. 35, P = .09).

Knowledge retention, as determined by an end-of-semester quiz that was the same as the baseline quiz students completed, showed both groups performed similarly. The increase in knowledge retention [TMM: 67% (50, 83) vs. EQV: 83% (50, 83)] from baseline (33% in each group) was high in both groups, so some knowledge was retained regardless of quiz presentation.

The response rate on the preference survey administered to the EQV group was 74% (n = 20/27). A high percentage of students in EQV group felt more engaged (n = 14/20, 70%) with the pre-class videos, while 15% (3/20) felt similarly engaged and 15% (3/20) less engaged. Of those 20 students, when asked if they had a preference for either EQV or TMM overall and for class preparation, 19 (95%) and 12 (60%) indicated they did, respectively. Of those indicating a preference, 58% (11/19) preferred EQV to TMM overall, and 75% (9/12) preferred EQV to TMM for class preparation.

DISCUSSION

The results of this study show that embedding quizzes in pre-class videos only partially produced the effects anticipated. Students did not perform better on pre- or post-class quizzes or retain more knowledge overall; however, students exposed to embedded quizzing were more likely to view pre-class videos, showing that the embedding of the quizzes caused students to utilize the pre-class content. Results of this study regarding student perception of the benefits of answering quiz questions embedded within videos are similar to those of previous studies investigating embedding of quizzes in pre-class videos.[16-18]

In a study by Rice and colleagues, two different groups of students performed better on quizzes assessing content presented in embedded-quiz videos (POD 2014 = 80.98; MKT1022 = 49.69) than on content presented in videos with quizzes at the end (POD 2014 = 75.22; MKT1022 = 34.06) or no quizzes (POD 2014 = 65.23; MKT1022 = 26.88), though formal statistical analysis was not conducted.[16] Another study found that student performance was significantly improved when quiz questions were embedded in an interactive video lesson in comparison with conventional video and quiz format, in which quiz questions were presented at the end (8.98 vs. 8.68, p < .001). This comparison was made using the same students, with the interactive videos being used during the first half of the semester and the traditional videos being used in the second half; as such, content was different across video formats.[17] Alternatively, Ketsman and associates conducted a randomized study in a physics course with a similar design to the current report.[18] Students did not perform better on quiz questions whether they were embedded or presented at the end of the video (66 vs. 72, p = 2). While discrepancies between methodology and results in previous studies make determining the efficacy of embedding quizzes into videos difficult, all three studies concluded embedded quizzes increased student engagement.

Strengths of the current study include that it was randomized, students had similar baseline assessment scores, and this appears to be the first assessment of retention of information over multiple weeks of a semester using embedded-quiz videos. Limitations include small sample size and restrictions due to the technical capabilities of the video recording software. More specifically, students in the EQV group were unable to see, and therefore answer, quiz questions when viewing videos on mobile phones or iPads/tablets. Despite efforts by faculty to prevent students in the EQV group from using mobile devices or tablets for video viewing, some students (two for one week of videos, one for remaining two weeks of videos) chose to watch videos on mobile devices, and their grade was affected by an inability to answer quiz questions. Additionally, another limitation of this study is short duration of the intervention as results and outcomes were only assessed from weeks five through seven of the semester. Student use and preferences may be influenced by longer durations and this could be studied in future research. Finally, total student views as an outcome counted any time a student accessed a video, including short viewings; however, the higher number of total views in the EQV group is supported by the fact that students in that group also viewed a much higher number of minutes of videos.

Faculty learned several lessons while implementing and researching this educational intervention. A persistent lesson learned throughout this study related to the limitations of technological software. Faculty must anticipate technological limitations and attempt to ameliorate these during the study design phase of research. Even so, students may not respond as predicted; evidence of this is shown by students in the EQV group choosing to watch pre-class videos on phones or tablets despite being told to use computers so the quiz questions would appear. Faculty anticipated this issue and believed instructing students verbally and in writing would be sufficient to circumvent this technological limitation. However, some students in the EQV group disregarded these instructions, suggesting that another form of notifying students of technological limitations may be warranted. After the completion of this research project, the librarian desired to increase students’ engagement with the videos due to the overall low utilization. The librarian re-recorded the longer pre-class videos by dividing the content into shorter, more manageable chunks of information for students in future iterations of the course.

From educators’ perspective, utilization of embedded quizzes as opposed to LMS quizzes requires some extra time and effort. Time dedicated to the development of pre-class videos, handouts, and quiz questions was substantial, but already planned due to the nature of the flipped course combined with the traditional delivery of content and assessments. To embed the quiz questions into the videos required about five minutes per video. However, while embedding of the quizzes into the videos was simple, the extrapolation of student performance on the quizzes was not seamlessly integrated across the video recording software and LMS. This increased the workload from faculty as students’ grades had to manually be entered into the LMS.

An unexpected finding of this study was that despite low video usage overall, the TMM group still scored high on pre-class quizzes. Each video had an accompanying handout available via the LMS, and all students had access to these handouts despite group assignment. It is possible students in the TMM group utilized handouts to take the quizzes, which may explain their high performance despite low views of pre-class videos. Another interesting finding was the reduction of score from pre- to post-class quiz grade in the EQV group, from 93% to 87%. Potential explanations include that the quizzes were different or that there was a benefit seen from the timing of the quiz questions embedded in the videos allowing for immediate recall of that information via the subsequent question. Additionally, the positive student opinion of the embedded quizzing experience despite no difference in grades was an unexpected finding of this study. While taking embedded quizzes did not improve students’ performance or knowledge retention, students in the EQV group felt more engaged with pre-class videos, and some of those preferred taking embedded quizzes compared with taking them in the LMS, which suggests that the embedded quizzes had the desired effect of engaging students with videos upon which they might otherwise lose focus. This information, combined with the fact that students demonstrated that they were more likely to watch videos with embedded quizzes, suggests using the embedded quizzes as a formative rather than summative assessment technique may have value for students’ learning. Finally, using embedded quizzes for low-stakes formative assessment may be more appropriate given the technological limitations of the video recording software. Practice testing, or retrieval practice, has been proven to improve student learning compared with other study methods.[21,22] Modifying video-embedded quizzes from summative assessment to low-stakes formative assessment with immediate feedback may be a unique mechanism for employing practice testing in the classroom.

CONCLUSIONS

Administering quizzes embedded in pre-class videos in a DI flipped class did not affect end-of-semester knowledge retention or immediate students’ performance. However, students were significantly more likely to view the pre-class videos when quizzes were embedded. Students in the EQV group, who experienced both assessment methodologies at different times during the semester, preferred embedded quizzes overall and for class preparation and felt more engaged with the pre-class videos. Due to the small sample size of this study and limited published literature, further studies are needed to determine implications of embedded quizzing in pre-class preparation materials in the flipped classroom setting.