What millennial medical students say about flipped learning.

Flipped instruction is gaining popularity in medical schools, but there are unanswered questions such as the optimum amount of the curriculum to flip and whether flipped sessions should be mandatory. We were in a unique position to evaluate feedback from first-year medical students who had experienced both flipped and lecture-based courses during their first semester of medical school. A key finding was that the students preferred a variety of different learning formats over an "all or nothing" learning format. Learning format preferences did not necessarily align with perceptions of which format led to better course exam performance. Nearly 70% of respondents wanted to make their own decisions regarding attendance. Candid responses to open-ended survey prompts reflected millennial preferences for choice, flexibility, efficiency, and the ability to control the pace of their learning, providing insight to guide curricular improvements.

Introduction

Most of today’s medical students are millennial or Generation Y learners.1 Millennials expect choice, flexibility, speed, and efficiency.2 They are usually attentive learners if the format is on their own terms,3 and they can control the pace, sequence, and content.4 Millennials are technology savvy and often prefer to learn by trial and error5 and by doing,4 both major components of the flipped classroom.6

In a flipped classroom, typical lecture and homework elements are reversed; recorded lectures, vodcasts (video podcasts), or readings are viewed by students prior to class, while in-class time is devoted to more student-centered activities.6 Various medical schools have reported their experiences in flipping courses, disciplines, and even entire curricula.7–13 While other medical schools have compared student satisfaction with flipped instruction to cohorts from previous years who experienced traditional instruction,7,11 we had the ideal environment to obtain input regarding both strategies from a single cohort. This cohort experienced a pilot flipped course followed by four traditional courses in their first semester of medical school.

Evaluation of instructional methods should address learner beliefs and preferences.14,15 With this in mind, the aim of this study was to identify stakeholder preferences prior to curricular changes, with the goal of increasing opportunities for student learning. The following research questions were posed: Do first-year medical students (MS1s) prefer flipped or lecture-based formats or a combination of approaches? Which format do they think helps them learn better? What are students’ obstacles to mandatory attendance for flipped activities? Are there additional teaching strategies that would aid their learning styles?

Methods

Participants and setting

The first course of the year for the class of 2019 (MS1s) was a pilot 3-week anatomy course, Basic Structural Foundations, in which 75% of the content was flipped and the other 25% was lecture-based with embedded activities. For flipped sessions, students viewed vodcasts containing interactive quizzes prior to coming to class for non-mandatory practice activities. The next four courses were lecture-based, ~70% of these with built-in activities, and a small percent (5%) flipped with vodcasts. Activities for flipped sessions and lecture-based sessions varied widely, from drawing anatomical structures to games to case presentations (McCoy et al, manuscript in preparation).

Development of the survey instrument

An original, four-item, interactive, electronic Qualtrics™ survey (Figure S1) was used to gather student feedback on flipped learning. Two questions at the beginning of the survey addressed age range and gender.

Data collection

MS1s were surveyed in January 2016 after completing the five courses described earlier. Survey data collection involved an email solicitation containing a clickable link to the online survey. Survey participation was voluntary and anonymous, and there were no rewards offered for completing the survey. Completion of the survey was deemed consent. The A. T. Still University Institutional Review Board (IRB) deemed the study exempt from IRB reporting requirements for human subjects research because no collected data associated directly through identifiers linked to participants and was conducted via anonymous surveys.

Data analysis

We used open- and axial coding16 to analyze students’ responses to open-ended survey prompts.

Results

Survey response rate and demographics

A total of 104 students (91.0% response rate) participated in the survey. There were 54 male and 50 female respondents (65 aged 23–25 years, 31 aged 26–30 years, 7 aged 31–35 years, and 1 aged 37 years).

Do MS1s prefer flipped, lecture-based formats or a combination of approaches?

A major unanswered question for educators implementing the flipped method is the optimum amount of the curriculum to flip. We probed this topic with the following interactive survey question: “Slide the scale to indicate your preferred large group format”. The preferred large group format for 18% of respondents (n=94) was 50% flipped/50% lecture based (Figure 1). There was tremendous variation in preferred pedagogical approach for the rest of the students. Indeed, the histogram in Figure 1 shows a peak at 50% flipped/50% lecture based, with the remaining responses distributed almost equally on either side.

Figure 1

Summary of student responses to the prompt, “Slide the scale to indicate your preferred large group format. Slide the dial to any % flipped, from 0% to 100%.”

Notes: Flipped = typical lecture and homework elements of a course are reversed; video lectures are viewed by students at home before the class session, while in-class time is devoted to exercises, projects, or discussions (Association of American Medical Colleges, 2013). Lecture-based = lectures with activities built into some presentations.

Twenty students provided optional comments regarding their preferred large group format. When these comments were categorized using open and axial coding,16 six key themes emerged: Pace, Organization, Discipline-specific, Schedule/Echo360™ (lecture capture technology), Suggestions to enhance lecture-based with activities, and Independent study (Table 1). Some comments reflected perceived drawbacks to the flipped method in terms of Pace and Organization. For example, one comment exemplified the millennial desire to control the pace of their learning: “People work at different paces, and flipped classrooms encourage rushed learning, especially when people blurt out answers, it makes you feel behind and inadequate” (theme Pace). Comments that are coded with Discipline-specific and Schedule/Echo360 provided suggestions for selective use of the flipped method and illustrate the millennial preference to control the content and sequence of their learning, including “This really depends on the subject matter. For me, I like flipped for anatomy and micro courses as there is simply a bunch to memorize and the podcasts/vodcasts do a good job of this tedious stuff and then in class activities for what we learned. However, courses like physiology, biochemistry, genetics, I prefer regular lectures given in class” (theme Discipline-specific) and “Enjoy flipped classroom, however would prefer if they occur earlier in the week so there is enough time over the weekend to prepare for those classes” (theme Schedule/Echo360). Three comments that are coded with the theme Suggestions to enhance lecture-based with activities were supportive of preview videos (short vodcasts on basic concepts), which were provided in advance of some biochemistry and genetics lectures. A few students indicated their preference for other methods of receiving information (theme Independent study), including reading.

Table 1

Summary of student responses to optional comments regarding preferred large group format

Theme	Specific comments
Pace	Flipped classrooms rush learning. People work at different paces, and flipped classrooms encourage rushed learning, especially when people blurt out answers; it makes you feel behind and inadequate.I feel like the flipped classroom does not work for me. It is usually group-based activities and the person who leads the group is very vocal and understanding of the material. I do not learn best being overshadowed in the content by my peers.
Organization	Flipped-based classes often are too unorganized due to large group sizes and other students blurting out answers, thus making them ineffective.I personally feel that many times with the in-class activities, there are too many students all trying to do the same thing, which makes it hard for individual students to learn and benefit from the activity.
Discipline-specific	I really enjoy activities for anatomy and embryology because it can be difficult to get proper understanding of how things work together.This depends on the material. For anatomy, the flipped works well, but for more dense topics in a subject like neurology, then flipped and lecture-based are well in 50/50.I think it depends on the content. For instance, dense topics such as biochemistry and physiology are probably best in lecture-based format. However, topics such as anatomy are best taught through activities in the flipped format.This really depends on the subject matter. For me, I like flipped for anatomy and micro courses as there is simply a bunch to memorize and the podcasts/vodcasts do a good job of this tedious stuff and then in-class activities for what we learned. However, courses like physiology, biochemistry, and genetics, I prefer regular lectures given in class. I do like the idea of these courses providing podcasts/vodcasts in addition to regular in-class lectures to explain harder concepts, but I do not want in-class activities for these subjects. Again, in-class activities for anatomy, micro, and clinicians are the only subjects I would like to see. I would also like to see more basic science lectures (biochemistry, genetics, pharm).
Schedule/Echo360™	Enjoy flipped classroom, however would prefer if they occur earlier in the week so there is enough time over the weekend to prepare for those classes.The only concern in having flipped is if we get the chance to preview the material.Will these exercises, projects, and discussions be ECHOed? I think a big reason why people choose not to go to class and to watch ECHO is because you can study at your own pace and study when you are at your sharpest. Not everyone is a morning bird. I would then recommend that the activities be in the middle of the day so that morning people and evening people can get the most out of it. (Also fits under Pace)The flipped classroom format is difficult sometimes to prepare for in time. I like it when there are activities worked into the lectures or using the last few minutes of lecture for an activity. That helps to cement the information of the lecture there and then and clarifies any gray areas. (Also fits under Lecture-based with activities)
Suggestions to enhance lecture based with activities	Keep things the way they are, may be a little more interactive lectures.I prefer receiving the lecture material in class and doing some built-in exercises to help the material commit to memory.When the podcasts given before lecture were short intros they were effective versus large sums of information that lecturers then just skipped over during lecture because they were “already discussed”.Some contents for the preview videos were good, while others were extra background information I did not find necessary. I did like the idea of the preview video.I would rather prefer the concept of still having lecture-based material presented, as it is this year, with the additional of short preview videos/lecture ahead of time.
Independent study	I prefer to view the lecture slides on my own and then come to class only to ask questions if I am confused on a certain topic. Of course, the main drawback to this is that the lecture slides must be absolutely 100% accurate and include everything that will be tested on. I prefer to not have podcasts if the lecture slides are accurate to read off of.I usually just outline my lectures from the powerpoint presentations and then listen to podcasts as necessary. Usually I do not physically go to large group unless its mandatory.I really dislike lectures, they are too passive, and for me, the challenge becomes less about the material and more about trying to discern which content this particular professor finds most important. I also feel that the only way to create sustained knowledge is to spend time reading and making connections with the material before interacting with a professor in a large group setting. I would prefer to have our time spent with professors discussing the topics, posing questions, and working through activities that connect that topic to other aspects/subjects required for the board exam. I would rather use the professors as a resource for deeper understanding of the material instead of reading their powerpoints to us. That just promotes Echoing simply because it is more efficient and you can turn 8 hours of lecture into 4–5 hours. Personally I gain greater sustained knowledge if I were to spend 4–5 hours reading.

Note: Echo refers to Echo360™ lecture capture technology.

Which pedagogical format do MS1s think helps them learn better?

The follow-up survey prompt was “Best learning format. Disregarding your preferred large group format above, which of these methods leads to better course exam performance for you?” Of the 17 students who selected 50% lecture/50% flipped as their preferred large group format, six thought that flipped formats led to better course exam performance, five thought that lecture led to better course exam performance, and six selected “other”, explaining that either a combination or some alternate method (eg, study groups) worked best. For the remaining students, ie, those who selected a combination other than 50% lecture/50% flipped, 75.9% perceived that their selected method led to better exam performance, 19% perceived that the opposite of their preferred modality led to better exam performance, and 5.1% chose “other”, explaining that either a combination or some alternate method led to better exam performance.

What are MS1 obstacles to mandatory attendance for flipped activities?

Another unanswered question regarding flipped learning is whether in-class activities should be mandatory. When the preclinical curriculum was flipped at one medical school, student attendance and engagement were lower than expected,13 which was partially attributed to non-mandatory attendance.17 Our medical school does not currently require attendance for lecture-based and flipped sessions, but lectures are recorded via lecture capture.

In this study, 68.7% of respondents thought that in-class practice activities for flipped sessions should be non-mandatory. Their responses to the follow-up prompt, “Please explain your choice” coded into seven key themes: Learning style, Attitude/Preferred learning conditions, Not efficient, Choice, Lack of preparation/time, Situation-dependent, and Pace (Table 2). Many of these themes and associated comments aligned with millennial expectations for choice, flexibility, efficiency, and the ability to control the pace of their learning.2 For example, the millennial desire for choice was exemplified by the comment “People should have the choice if they want to use in-class activities or their own activities to strengthen their understanding of material” (theme Choice). The millennial desire to control the pace of their learning was exemplified by the comment “I like practice activities but I do sometimes feel like I take a backseat in group work. Other people come up with answers and I feel like I do not have as much time to work through problems on my own” (theme Pace). Two enlightening comments under the theme Attitude/Preferred learning conditions were “When people are required to attend but do not actually want to be there, it ruins it for those of us who do want to attend” and “I like when it is less crowded.”

Table 2

Summary of student responses to the open-ended prompt, “Please explain your choice”, that followed “For flipped sessions, should in-class practice activities be mandatory?”

Theme	Student explanation for selection of non-mandatory attendancea
Learning style	Reading assignments are more important.Not everyone can learn this way (8).Some people do not learn as well from activities, so their time might be better spent reviewing on their own (also fits under Not efficient) (2).I do not think they are necessary and I think that people can learn as much if not more if they do the activities themselves.
Attitude/Preferred learning conditions	When people are required to attend but do not actually want to be there, it ruins it for those of us who do want to attend.Activities would only be useful if the participant is willing to be there. Making attendance mandatory will limit some benefit.It is the student’s loss if not participating. Learning sessions should never be mandatory if ECHOed, we are in professional schools and student should be selected who are believed to uphold this professionalism.I like when it is less crowded and I think we can gauge where we need help. Forces us to be more self-accountable.
Not efficient	This would require far too much mandatory time if there were also outside lectures to watch. Very little time would be left for self-study.The activities often consume a lot of time that could be used more efficiently (3).Because it is still beneficial on Echo.
Choice	If people do not want to participate, they should not get the benefits of participating.People should have the choice of if they want to use in-class activities or their own activities to strengthen their understanding of material (also fits under Learning style).If the student does not feel that he or she needs additional practice, there should be no requirement to attend.Some people find them quite useful, while others can get more confused by not doing it “their way.” To me, flipped classroom activities are “study” if you do not find the particular one to be helpful, it may feel a waste of time (also fits underLearning style and Not efficient).It would of course be most beneficial to be in class for these activities and I think most students will gravitate toward attendance. But, I feel that there may be an occasion where you may have to miss class for an undisclosed reason or because you feel that you would prefer to learn that particular topic in a different manner. We are all professionals and we should be seeking the strongest level of understanding, thus the benefits of coming to class should stand on their own. It should not be “forced”, creating a fluid style of teaching will best suit all learning styles while still emphasizing what the school finds to be the most efficient way (also fits under Learning style).
Lack of preparation/time	Sometimes people are not prepared for the activities ahead of time and so do not get anything out of the session if they go.I like that classes are optional and if I am not prepared at a certain time there is no need to rush.Cannot attend them all.
Situation-dependent	Sometimes I do not find the large group practice activities helpful. My participation greatly depends on the professor in charge.They should be optional. For some of the subjects that I have already studied, the flipped sessions are not worth the time (also fits under Not efficient) (2).We are paying 50,000/year for medical education. Students will use their time as they deem most valuable to achieve academic success. Thus, in-class practice activities must have broad appeal and high yield to attract students to attend these sessions.
Pace	I still prefer going at my own pace.I like practice activities but I do sometimes feel like I take a backseat in group work. Other people come up with answers and I feel like I do not have as much time to work through problems on my own.

Notes:

These comments were from students who selected non-mandatory. Echo refers to Echo360™ lecture capture technology. Parentheses refer to the number of additional similar comments obtained from respondents.

Nearly one-third of respondents (31.3%) indicated that flipped learning sessions should be mandatory. Their responses to the follow-up prompt “Please explain your choice” coded into six key themes: Motivation to attend, Motivation to stay on top of material, Application of knowledge, Promotes learning, Activity enhanced with increased participation, and Situation-dependent (Table 3). Motivation to attend, Motivation to stay on top of material, and Situation-dependent were the most common themes. An unexpected comment that was coded with Motivation to attend was “Non-mandatory makes the material feel unnecessary.” Other valuable perspectives included “For many activities having more participants makes the activity more enriching (such as competitive games)” (theme Activity enhanced with increased participation). Well-structured activities, a critical component of a successful flipped class, were important to both group of students, those who supported non-mandatory attendance (Table 2, theme Situation-dependent) and those who supported mandatory attendance (Table 3, theme Situation-dependent).

Table 3

Summary of student responses to the open-ended prompt, “Please explain your choice”, that followed “For flipped sessions, should in-class practice activities be mandatory?”

Theme	Student explanation for selection of mandatory attendancea
Motivation to attend	I think the activities are helpful, and if they were mandatory and given a specific allotment on time, I would be more inclined to go. (Rather than sitting in a large group lecture and not knowing if there were going to be practice activities or watching the lecture online and missing out on the activities.)To feel the obligation of being there. Non-mandatory makes the material feel unnecessary (2).I think if activities counted for part of the course grade, then I would be more likely to do them. But the activities have to be thought out very carefully because otherwise it might take time away from other studying.
Motivation to stay on top of material	Ensures that everyone actively participates, are prepared, and on top of their video lectures (4).This will allow me to make sure I have faculty available to ask questions and would allow me to make sure I complete the sessions in an orderly and timely fashion.
Application of knowledge	Primary knowledge is easy to learn independently, and practice with oversight instruction would confirm my knowledge (1).There is a need to future physicians of being able to see before clinic how cases, etc., are.
Promotes learning	Students are less likely to benefit from the activities if they do not attend (2).Interactivity better promotes learning.This is where content is solidified and confusion is rectified. It is vital.
Activity enhanced with increased participation	For many activities, having more participants makes the activity more enriching (such as competitive games).If they are not mandatory, then students will be less likely to attend and will not get the full benefits of having a flipped classroom. Also, it is a better learning experience when more students are present to participate (also fits under Motivation to attend)
Situation-dependent	If lengthy enough and worth attending, I believe in-class practice activities can be very beneficial.They must be well-structured, not a free for all, do whatever station you want.Yes, I think many in-class practice activities should be mandatory so that the essential content remains consistent for all students. However, there may be some subjects that may be more straightforward or review and there may be a significant number of students who may not benefit from additional practice activities.Especially if material from activities will be on test.

Notes:

These comments were from students who selected mandatory. Parentheses refer to the number of additional similar comments obtained from respondents.

Are there additional teaching strategies that would aid their learning styles?

In response to the prompt, “Are there additional teaching and learning strategies that would aid your learning style?”, 11 categories emerged: Incorporation of more practice questions, including clicker questions (16% of respondents); Larger formative quizzes (12% of respondents); Review sessions (4% of respondents); Cumulative exams (2% of respondents); Independent study (6% of respondents); Group study (2% of respondents); Vodcast transcripts (4% of respondents); Board preparation including professors “referencing First Aid pages” (4% of respondents); Inclusion of external vodcast sources (4% of respondents); Mandatory individual small group preparation so that students “can rip through as many cases as we can in the time given” (2% of respondents); and Large group strategies including “organized PowerPoints”, “tons of examples”, “games”, “more models”, “connecting lecture material”, “small groups where we draw physiologic processes with peers to demonstrate our understanding”, and “presentations that touch on a variety of learning styles” (44% of respondents).

Discussion

While student preference does not necessarily equate to what approach is best for student learning, millennials are more likely to be attentive learners if the format is on their own terms.3 In addition, health science student engagement has been shown to positively influence academic achievement.18 As such, stakeholder feedback regarding the two most implemented pedagogical approaches (flipped or lecture-based), including student suggestions for optimizing them, will be considered during curricular improvements.

One of the most surprising results was the relatively large number of students who preferred >50% lecture-based classes. This finding was unexpected for several reasons: the high ratings MS1s gave the pilot flipped course, and millennial preferences for collaborative learning1 and learning by doing.4 These student preferences may have been influenced by the fact that most lectures had some active learning component or that lecture was perceived to be more efficient than the flipped method.19 A small percent of the respondents wanted 100% lecture, a finding that other medical schools have reported.7,11 In addition to the insights provided by our students, possible explanations for lecture preference include undergraduate experience, rock concert “event-ness” making the material memorable,20 the auditory-dominant learning mode of high-risk medical students,21 and students’ perceived misalignments between active learning and national board exams.19 Methods to engage students with a preference for lecture should be investigated, including those suggested by our students under Large group strategies (“Results” section).

Another unexpected result was that large group learning format preference did not always align with perceptions of which format led to better course exam performance. Rigorous studies are needed to determine if discrepancies between preferred learning format and actual outcomes are significant.

Student attendance requirements are the subject of ongoing debate in the medical education community. The availability of lecture capture technology has increased the number of students who choose not to attend class. For students in this study, obstacles to mandatory attendance were largely focused on choice, flexibility, speed, efficiency, and control of the format and pace of their learning; all well-known millennial expectations.2–4 While our faculty discuss the merits and drawbacks of particular attendance policies, we are incorporating MS1 suggestions to make flipped sessions efficient, high yield, and of broad appeal, and are providing enough time in the curriculum for students to prepare for these sessions. The hope is that these efforts will motivate students to attend and potentially increase their learning.

Moffett22 suggested that a flipped learning approach does not have to be “all or nothing,” and only a minority of our students preferred all or nothing. In fact, the range of individual preferences for pedagogical approach was striking. Millennials were raised in an entertainment-focused, multi-media environment; as a result, they have a low tolerance for boredom.23 Lack of pedagogical variety has the potential to cause boredom and to negatively impact students who do not perceive that they learn with the pedagogical method being implemented. In this study, the need for faculty to accommodate varied learning styles was evident throughout survey responses, lending further support to the important role of variety in the medical school curriculum.10,24,25

A recent study provided evidence that the benefits of the flipped model are simply due to the active learning component.26 As such, our results may have been influenced by the fact that ~70% of our lecture-based large group presentations have an active learning component.

Limitations

This research is limited by the institutional and cultural contexts in which it was conducted. Surveys were collected anonymously to reduce the likelihood of response bias. We were not able to characterize non-responders, who may or may not have had different perceptions of pedagogical approach. These students were in their first year of medical school, which may have influenced perceptions. The results of this study might not be generalizable to other educational programs or cultures. In an attempt to apply these methods and findings in a different context, investigators should consider the specific constraints, type of learning, outcome measures used, and the natural environment of the study setting.

Flipped Preferences Survey

Abbreviations: AAMC, Association of American Medical College; SOMA, School of Osteopathic Medicine in Arizona.