The Effectiveness of Flipped Classroom in Health Professions Education in China: A Systematic Review.

BACKGROUND: Flipped classroom has received much attention during the last few years in China, but inconsistent conclusions are made about the effectiveness this approach in health professions education. This review examined the findings of controlled studies published in Chinese in order to summarize the effects of the flipped classroom methodology. These studies focused specifically on undergraduate level of health professions students.
METHODS: A literature search was conducted using China National Knowledge Infrastructure (CNKI) and Wanfang Data Knowledge Service Platform in June 2019. No date restrictions were used. Peer-reviewed papers were reviewed and experimental studies were included if the study compared student outcomes using flipped classroom method versus traditional lectures. The revised version of the Kirkpatrick's model was applied to evaluate the effectiveness of the flipped classroom approach.
RESULTS: In total, 934 articles were obtained. Among them, 235 articles were full text reviewed and 59 met the inclusion criteria. Most of the student agreed that flipped classroom strategy was very helpful improving their self-learning ability, problem solving ability, teamwork, and communication skills, but reported increased workload and less-efficiency compared with traditional lectures. However, the majority of studies (n = 52) showed positive findings of the flipped classroom on changes in knowledge and skills, and 3 in negative. Among these 59 studies, there was little evidence of changes in behavior, organizational practice and patient outcome.
CONCLUSIONS: Better design with reduced content should be considered for instructors to promote learning effectiveness. Long-term effects of flipped classroom should be considered in future studies.

Background

As educators adjust their instructional approach to improve teaching effectiveness and encourage active learning, the “flipped classroom” has become a popular strategy in China. In the flipped classroom (FC), lectures are replaced by pre-class activities, typically viewing online videos, and class time is devoted to interactive activities and discussions. The hypothesis is that FC enables students to learn outside the classroom and promotes active learning and engagement inside the classroom.[1]

For a teaching strategy to be considered successful, there must be evidence that students’ learning is enhanced. Supporters of FC methodologies suggest that academic performance is improved when utilizing FC compared with traditional lectures.[2] For instance, Hew and Lo demonstrated that the FC is more effective than traditional classroom in terms of student achievement after analyzing 28 comparative studies.[3] In addition, learning theorists opine that FC strategy follows a learner-centric teaching philosophy, and reframes the faculty-student relationship.[4] More specifically, FC allows students to access recorded lectures and other materials as often as they desire and use class time for interaction and application of content.[5] However, studies showed that the effectiveness of FC may be specific to the content being taught. For example, Ma’s study indicated that lectures should be the major teaching methodology in teaching the theoretical knowledge of biochemistry.[2] Some commentaries have raised concerns about the validity of the assessment methods which evaluate the effectiveness of the FC approach. According to Moffett, educators should choose the appropriate assessment method to evaluate the effectiveness of flipped classroom.[6] As more educational practitioners adopt the FC approach, it is urgent to explore the effectiveness of FC and identify the key factors as well as practical problems that contribute to its effectiveness.

Existing reviews, although limited, suggest that the effects of the flipped classroom on student learning outcomes are inconsistent when compared to traditional lectures.[7-9] Chen et al analyzed 9 studies in medical education and found mixed results on student knowledge and skill changes using the flipped classroom compared with the traditional lecture.[7] Gillette et al included 6 studies in pharmacy education and found mixed evidence on student outcomes between lecture and flipped classroom.[8] And Evans compared 24 studies in health professions education and identified 17 that showed a positive effect on student academic outcome when using the FC instead of traditional classroom teaching.[9] It is difficult to generalize about the effects of FC on student achievement due to the limited number of included studies. Further, the search strategy of published systematic reviews that examined the effect of flipped classroom versus lecture have been restricted to studies published in English. While literature on the use of the FC in health professions education has grown rapidly over the last few years in China, the research in Chinese will be able to include more information on the effect of FC compared to lecture in health professions education in the Chinese context. The purpose of this review was to systematically investigate the studies associated with the effectiveness of the FC in health professions education in China. This review was guided by the 2 research questions: (i) What is the scope of the published research on FC in health professions education in China? (ii) What are the effects of the FC in health professions education in China?

Methods

Literature search

This systematic review was carried out based on the PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analysis) guidelines.[10] Since this is a literature review, the study does not require ethical approval. We conducted unique searches in 2 Chinese electronic databases: China National Knowledge Infrastructure (CNKI), and Wanfang Data Knowledge Service Platform. The search Boolean was (“flipped classroom”) AND (“health professions education”) to capture all relevant article suggestions. Search was limited to peer-reviewed scholarly sources. Research reported in languages other than Chinese was omitted from this review. No date restrictions were used. This literature search was conducted and completed by the end of June 2019. Articles fell into the time span from the first paper on FC published in 2014 through June 2019. The search strategy was designed with the help of a research librarian from the university.

Review procedure

Figure 1 summarizes the review process and number of articles after each stage. After excluding duplicates, the first 2 authors reviewed the abstracts separately and discussed their findings. If they agreed, the article followed the process; if not, the full article was checked and discussed before being selected. When agreement on the included articles was reached, full articles were reviewed independently by the first 2 authors.

Figure 1.

Summary of the review process.

After the articles were identified, the 2 authors (SL and XL) reviewed each article in detail using a rating agreement developed to record information (Table 1). The rating protocol was adapted from Chen et al.[7] Articles which measured the effect of other instruction methods, such as TBL and PBL combined with FC, were excluded. In addition, articles were excluded if no effects of FC were measured. The measure of effectiveness of the FC was categorized based on modified Kirkpatrick’s evaluation model.[11] Based on this model, studies that used controlled studies were further analyzed for effectiveness of the FC to avoid bias. The 2 authors compared their decisions on all inclusions and resolved discrepancies in judgment by discussion. Finally, a total of 59 articles were included in the final analysis.

Table 1.

Variables and categories of rating protocol.

Variables	Pre-determined categories
Authors	Not applicable
Year published	Not applicable
Journal	Not applicable
Targeted sample	Undergraduate level of health professions education (Year 1-5)
Methodological design study type	Post-course designs
	Non-Randomized controlled studies
	Randomized controlled studies
Kirkpatrick’s classification	Perceptions of intervention
	2a. Changes in attitude
	2b. Changes in knowledge and skills
	3. Changes in behaviors
	4a. Changes in organizational practice
	4b. Changes in patient outcome

Measures of effectiveness

The studies’ measurement of effectiveness was evaluated according to the revised version of the Kirkpatrick’s model[7]—the 6 hierarchical levels of classification of the outcomes are: (1) learner perception to the intervention; (2a) changes in learner attitude; (2b) changes in learner knowledge and skills; (3) changes in learner behavior; (4a) changes in learner organizational practice; (4b) changes in patient outcome. All 59 articles reported the intervention outcomes. Of the 59 studies, 52 tested the effectiveness of the FC in terms of both learners’ perceptions and knowledge and skills gained. Six studies focused only on the changes on students’ knowledge and skills, while 1 study focused only on the students’ satisfaction with the FC (Table 2).

Table 2.

Summary of controlled studies on FC in health professional education in China (n = 59).

Authors	Year	Topic	Design	Treatment	Level	Sample	Kirkpatrick Measure of Effectiveness	Findings on effectiveness in Kirkpatrick Framework
Li[12]	2018	Nephrology	Randomized controlled study	FC: pre-session online videos and pre-class questions, online tutorial and group discussion; class time devoted to problem solving and group discussion.Control: traditional lecture	Fourth-year medical students (Clerkship)	Treatment (n = 36)Control (n = 36)	2b: Test (MCQ, and short answer questions); Teaching Quality Assessment Survey (self-reported on skills changed)	FC group performed better (P < .05); FC improved learning ability, clinical thinking and self-learning.
Wang et al[13]	2017	Stomatology	Non-randomized controlled study (selected by groups)	FC: pre-session online videos and group discussion; class time devoted to dental sculpting and group discussion.Control: traditional lecture and dental sculpting experiments	Second-year medical students	Treatment (n = 62)Control (n = 60)	1: Satisfaction survey2b: Test (MCQ), and sculpting skills	FC group performed better on both test and sculpting skills (P < .05); FC improved learning ability, learning efficiency and self-learning.
Ma et al[14]	2018	Physiology	Randomized controlled study	FC: pre-class orientation, and learning materials and assignment; in-class quiz and case study.Control: traditional lecture	Second-year medical students	Treatment (n = 30)Control (n = 30)	2b: Critical ThinkingDispositiong Inventory-Chinese Version	FC group performed better on systematic and integrated capabilities (P < .05)
Li et al[15]	2018	Stomatology	Non-randomized controlled study (selected by groups)	FC: pre-class orientation, and pre-session online videos, reading, PPTs, and assignments and group discussion; students presented their learning outcome by groups in-classControl: traditional lecture	Third-year medical students	Treatment (n = 60)Control (n = 60)	1: Satisfaction semi-structured interview2b: Test and self-learning ability assessment	FC group performed better on both test and self-learning ability assessment (P < .05); FC improved learning ability, teamwork, communication skills, learning efficiency, and self-learning.
Yang[16]	2016	preventive medicine	Non-randomized controlled study (selected by groups)	FC: pre-session online videos; class time devoted to discussion.Control: traditional lecture	Third-year nursing students	Treatment (n = 64)Control (n = 64)	1: Satisfaction survey2b: Standard test	FC group performed better on standard test (P < .001); FC improved learning ability, critical thinking, problem solving ability, teamwork, communication skills, learning efficiency, and self-learning.
Chai et al[17]	2018	Internal medicine	Randomized controlled study	FC: Learning group forming through WeChat, and pre-session online videos, case-study, pre-class quiz, and learning diary assignment; students presented their learning outcome by groups in-classControl: traditional lecture	Third-year medical students	Treatment (n = 30)Control (n = 30)	1: Satisfaction survey2b: Test (knowledge and case analysis)	FC group performed better on test (P < .05); FC improved learning ability and active learning.
Fang et al[18]	2018	Anesthesiology	Randomized controlled study	FC: pre-session online videos, learning questions and learning assignment; class time devoted to discussion.Control: traditional lecture	Medical students (cleakship)	Treatment (n = 31)Control (n = 31)	1: Satisfaction survey2b: Test (knowledge test and clinical skills test)	FC group performed better on both tests (P < .05); FC improved learning ability, learning efficiency and clinical skills ability.
Zhan et al[19]	2017	Public health	Non-randomized controlled study (selected by groups)	FC: pre-session online videos, PPTs and learning assignment; class time devoted to group discussion, PBL and community education.Control: traditional lecture	Medical students	Treatment (n = 212)Control (n = 218)	1: Post-intervention survey2b: Standard test (MCQ, short answer questions, and comprehensive questions)	FC group performed better on standard test (P < .05); FC improved self-learning ability, engagement, and teamwork. FC increased the learning load.
Bai et al[20]	2018	Surgery	Non-randomized controlled study (selected by groups)	FC: pre-session online videos, PPTs, online information, and learning questions; class time devoted to discussion.Control: traditional lecture	Third-year medical students	Treatment (n = 60)Control (n = 60)	1: Post-intervention survey2b: Test (MCQ)	FC group performed better on test (P < .05); FC improved self-learning ability, and increased peers communication and faculty-student communication. FC videos should be improved.
Ma et al[2]	2018	Biochemistry	Non-randomized controlled study (selected by groups)	FC: Learning group forming through WeChat and QQ, and pre-session online videos, PPTs, case-study, pre-class quizzes, and group learning and discussion; students presented their learning outcome by groups in-class aligned with active discussion.Control: traditional lecture	First-year medical students	Treatment (n = 60)Control (n = 60)	1: Post-intervention survey2b: Standard test (MCQ, judgment question, short answer questions, case analysis, and comprehensive questions)	FC group performed better on test (P < .05); FC improved learning efficiency, problem-solving ability, and increased peers communication and faculty-student communication. FC increased the learning load.
Pan et al[21]	2017	Nursing	Randomized controlled study (with pre-test)	FC: In-class group discussionControl: traditional lecture	Fourth-year nursing students (cleakship)	Treatment (n = 13)Control (n = 13)	2b: 14 standard tests after intervention with 6 months	FC improved the long-term memory.
Peng et al[22]	2016	Internal medicine	Randomized controlled study	FC: Learning group forming through QQ, and pre-session online videos, PPTs, case-study, and other learning materials; class time devoted to discussion.Control: traditional lecture	Medical students	Treatment (n = 90)Control (n = 90)	1: Satisfaction survey2b: Test (MCQ, judgment question, case analysis, and comprehensive questions)	FC group performed better on test (P < .05); FC improved self-learning ability, learning efficiency, and increased communication skills, problem-solving skill, and clinical skills.
Lang et al[23]	2016	Medical statistics	Non-randomized controlled study (selected by groups)	FC: pre-class leading questions and assignments; class time devoted to discussion based on the leading questions.Control: traditional lecture	Medical students	Treatment (n = 57)Control (n = 60)	1: Post-intervention survey2b: Standard test (knowledge questions, understanding questions, and comprehensive questions)	FC group performed better on test (P < .05); FC improved self-learning ability, and teamwork, and increased faculty-student communication. FC increased the learning load.
Song et al[24]	2016	Pathology	Randomized controlled study	FC: pre-session online videos, group discussion; class time devoted to discussion.Control: traditional lecture with case discussion.	Third-year medical students	Treatment (n = 30)Control (n = 30)	1: Satisfaction survey2b: Test (knowledge and case analysis)	FC group performed better on test (P < .05); FC improved self-learning ability, learning efficiency and teamwork, and increased faculty-student communication.
Xiao et al[25]	2017	Biochemistry	Non-randomized controlled study (selected by groups)	FC: Pre-class learning materials; in-class knowledge contest between groups and student group presentations.Control: traditional lecture	Third-year medical students	Treatment (n = 219)Control (n = 220)	1: Satisfaction survey2b: Test (MCQ, and short answer questions)	There is no difference between the two groups on test (P < .05). 99% FC group students satisfied with the FC method.
Xie et al[26]	2019	Radiology	Randomized controlled study	FC: pre-session online videos, PPTs, pre-class leading questions and assignments for group discussion; class time devoted to student group presentations and discussion.Control: traditional lecture with case discussion.	Fourth-year medical students (cleakship)	Treatment (n = 55)Control (n = 55)	1: Post-intervention survey2b: Standard test (knowledge questions, comprehensive questions and case analysis)	FC group performed better on test (P < .05); FC group students satisfied with the FC method. FC improved active learning.
Liang et al[27]	2016	Surgery	Randomized controlled study	FC: Learning group forming through QQ, and pre-session online videos, pre-class quizzes; class time devoted to clinical teaching and practice.Control: clinical teaching and practice.	Fourth-year medical students (cleakship)	Treatment (n = 42)Control (n = 42)	1: Post-intervention survey2b: Test (knowledge test and clinical skills) test)	FC group performed better on test (P < .05); FC improved active learning, learning efficiency and clinical skills.
Li et al[28]	2015	Obstetrics and gynaecology	Randomized controlled study	FC: Learning group forming through WeChat, and pre-session online videos; class time devoted to discussion.Control: traditional lecture	Medical students	Treatment (n = 60)Control (n = 60)	1: Self-assessment survey2b: Test (MCQ, judgment question, and case analysis)	FC group performed better on test (P < .05); FC group student got higher score on learning ability, learning efficiency and learning attitude.
Wang et al[29]	2018	Internal medicine	Non-randomized controlled study (selected by groups)	FC: Learning group forming through WeChat and QQ, pre-session online videos, PPTs, learning materials and pre-class quizzes, online tutorial and group discussion; class time devoted to group discussion with leading questions.Control: traditional lecture and experiment practice.	First-year medical students	Treatment (n = 90)Control (n = 90)	1: Post-intervention survey2b: Test (knowledge test and experimental test)	FC group performed better on both knowledge test and experimental test (P < .05); FC group students satisfied with the FC method; FC improved active learning, learning efficiency, self-learning ability, problem solving skills and communication skills.
Li et al[30]	2017	Clinical Nutriology	Non-randomized controlled study(selected by students’ ID)	FC: Learning group forming through QQ, pre-session online videos, PPTs, and pre-class quizzes, online case analysis and group discussion; class time devoted to group discussion with case study.Control: traditional lecture and case study.	Medical students	Treatment (n = 61)Control (n = 61)	1: Satisfaction survey2b: Standard test	FC group performed better on test (P < .05); FC improved active learning, self-learning ability, learning efficiency, teamwork and clinical skills.
Yu et al[31]	2019	Biochemistry	Non-randomized controlled study (selected by groups)	FC: pre-session online videos, PPTs, and learning materials for group discussion, online tutorial; class time devoted to student group presentations and discussion.Control: traditional lecture	Third-year nursing students	Treatment (n = 30)Control (n = 30)	1: Post-intervention survey2b: Standard test	FC group performed better on test (P < .05); 80% FC group students satisfied with the FC method; FC improved active learning and engagement.
Liu et al[32]	2018	Surgery	Randomized controlled study	FC: pre-session online videos, and pre-class quizzes, group discussion; class time devoted to group discussion and demonstration experiments.Control: traditional lecture and experiment practice.	Medical students	Treatment (n = 60)Control (n = 60)	1: Satisfaction survey2b: Standard test	FC group performed better on test (P < .05); 96.67% FC group students satisfied with the FC method while 78.33% control group students satisfied with traditional lecture.
Wu et al[33]	2016	Pediatric	Randomized controlled study	FC: Learning group forming through QQ, pre-session online videos, case study, and learning materials; class time devoted to group discussion with case study.Control: traditional lecture	Medical students	Treatment (n = 70)Control (n = 70)	1: Satisfaction survey2b: Standard test (MCQ)	FC group performed better on test (P < .05); FC improved active learning, but time-consuming and less efficiency.
Wang et al[34]	2018	Internal medicine	Non-randomized controlled study	FC: pre-session online videos, pre-class quizzes, and learning materials for group discussion; class time devoted to student group presentations and discussion.Control: traditional lecture	Medical students	Treatment (n = 57)Control (n = 63)	1: Satisfaction survey2b: Standard tests	FC group performed better on tests (P < .05); 89.47% FC group students satisfied with the FC method while 69.84% control group; FC improved active learning, learning ability, and communication skills.
Shan et al[35]	2016	Internal medicine	Non-randomized controlled study (selected by groups)	FC: pre-session online videos, pre-class quizzes, and group discussion; class time devoted to student group presentations and discussion.Control: traditional lecture	Medical students	Treatment (n = 32)Control (n = 32)	2b: Standard test	There is no difference between the two groups on test (P < .05).
Li et al[36]	2016	Medical Cell Biology	Randomized controlled study	FC: pre-session online videos, pre-class assignments, and group discussion; class time devoted to discussion, small group learning and experiments practice.Control: traditional lecture and experiment practice.	Stomatology and Nursing students	Treatment (n = 46)Control (n = 46)	1: Post-intervention survey2b: Standard test	FC group performed better on test (P < .05); FC group students satisfied with the FC method
Li et al[37]	2018	Medical Diagnostics	Randomized controlled study	FC: pre-session online videos, pre-class quizzes; class time devoted to discussion.Control: traditional lecture	Medical students	Treatment (n = 60)Control (n = 60)	1: Satisfaction survey2b: Standard tests (knowledge test including MCQ, definition, short answer questions, case analysis, and clinical skills test)	FC group performed better on both knowledge test and clinical skills test (P < .05); FC group students satisfied with the FC method; FC improved active learning, and learning ability.
Yin et al[38]	2017	Preventive medicine	Randomized controlled study	FC: pre-class learning materials, and pre-class quizzes,; class time devoted to discussionControl: traditional lecture	Nursing students	Treatment (n = 40)Control (n = 40)	1: Satisfaction survey2b: Standard test	FC group performed better on test (P < .05); FC group students were more satisfied with teaching (P < .05)
Nie et al[39]	2017	Anatomy	Non-randomized controlled study (selected by groups)	FC: pre-session online videos, pre-class assignments, and group discussion; class time devoted to student group presentations, discussion and experiment practice.Control: traditional lecture and experiment practice	Second-year Nursing students	Treatment (n = 50)Control (n = 50)	1: Satisfaction survey2b: Standard test (definition, short answer questions, and case analysis)	FC group performed better on test (P < .05); FC group students satisfied with the FC method
Liu et al[40]	2016	Internal medicine	Non-randomized controlled study (selected by groups)	FC: pre-class assignments, and group discussion; class time devoted to discussion with leading questions; after class case analysis assignments.Control: traditional lecture	Medical students	Treatment (n = 63)Control (n = 59)	1: Satisfaction survey and interview2b: Standard test (MCQ, and comprehensive questions)	FC group performed better on test (P < .05); FC group students were more satisfied with teaching; FC improved active learning, self-learning ability, problem-solving ability, teamwork and clinical thinking.
Li et al[41]	2019	Anatomy	Non-randomized controlled study (selected by groups)	FC: pre-session online videos, PPTs, and pre-class assignments and group discussion; class time devoted to student group presentations, and discussion.Control: traditional lecture	Third-year Medical students	Treatment (n = 38)Control (n = 39)	1: Satisfaction survey2b: Standard test	FC group performed better on test (P < .05); FC improved self-learning ability, teamwork and clinical thinking.
Ding et al[42]	2018	Dermatology	Randomized controlled study	FC: pre-session online videos, and experiment practice; class time devoted to student group discussion.Control: traditional lecture	Medical students	Treatment (n = 61)Control (n = 60)	1: Post-intervention survey2b: Standard test(MCQ, short answer questions, and case analysis)	FC group performed better on test (P < .05); FC improved self-learning ability, learning efficiency, teamwork, problem-solving ability and communication skills.
Peng et al[43]	2016	Medical Diagnostics	Randomized controlled study	FC: pre-session online videos, PPTs, and case-study; class time devoted to discussion.Control: traditional lecture	Medical students	Treatment (n = 96)Control (n = 96)	1: Satisfaction survey2b: Standard test (knowledge test and clinical skills)	FC group performed better on test (P < .05); FC group students were more satisfied with teaching (P < .05)
Liu et al[44]	2018	Basic Medicine	Randomized controlled study	FC: pre-session online videos, and pre-class assignments; class time devoted to discussion.Control: traditional lecture	Medical students	Treatment (n = 40)Control (n = 40)	1: Satisfaction survey	FC improved self-learning ability
Liu et al[45]	2018	Surgery	Randomized controlled study	FC: pre-session online videos, and pre-class quizzes, online tutorial; class time devoted to discussion.Control: traditional lecture	Second-year Medical students	Treatment (n = 48)Control (n = 48)	1: Satisfaction survey2b: Teaching effectiveness observation (self-learning ability, active learning, communication ability, and teamwork)	FC group students were more satisfied with teaching (P < .05); FC improved self-learning ability, active learning, communication ability, and teamwork.
Ren et al[46]	2018	Radiology	Randomized controlled study	FC: pre-session online videos, and pre-class quizzes; class time devoted to discussion.Control: traditional lecture	Fourth-year Medical students	Treatment (n = 25)Control (n = 25)	1: Satisfaction survey 2b: imaging diagnosis test	FC group performed better on test (P < .05); FC improved self-learning ability, communication ability, and imaging diagnosis skills.
Yang et al[47]	2017	Pathology	Non-randomized controlled study (selected by groups)	FC: pre-session online videos, PPTs, case-study, and pre-class quizzes; class time devoted to discussion.Control: traditional lecture	Nursing students	Treatment (n = 42)Control (n = 42)	1: Satisfaction survey2b: Standard test	FC group performed better on test; FC improved self-learning ability, teamwork, and clinical skills.
Gong et al[48]	2019	Clinical Skills	Randomized controlled study	FC: Learning group forming through WeChat, pre-session online videos, PPTs, and pre-class quizzes; class time devoted to discussion and clinical skills practice.Control: traditional lecture and clinical skills practice	Fourth-year Medical students	Treatment (n = 104)Control (n = 103)	1: Post-intervention survey2b: Standard test (Knowledge test and OSCE)	FC group performed better on both knowledge test and clinical skills test (P < .05); Over 90% students agreed that FC improved self-learning ability, provided more flexibility, and improved deep-learning.
Song et al[49]	2019	Anesthesiology	Non-randomized controlled study	FC: Learning group forming through QQ and WeChat, pre-session online videos, learning materials; class time devoted to discussion.Control: traditional lecture	Third-year and fourth-year Medical students	Treatment (Third-year, n = 89)Control (Fourth-year, n = 92)	1: Satisfaction survey2b: Standard test (Knowledge test and practice test)	FC group performed better on both knowledge test and practice test (P < .05); 98.88% FC group students satisfied with the FC method while 81.52% control group
Liu et al[50]	2017	Nursing	Non-randomized controlled study	FC: pre-session online videos, PPTs, pre-class quizzes, and group discussion; class time devoted to discussion.Control: traditional lecture	First-year and second-year Nursing students	Treatment (First-year, n = 31)Control (Second-year, n = 30)	1: Satisfaction survey2b: Standard test	FC group performed better on test (P < .05); FC improved active learning, self-learning ability, and teamwork.
Sun et al[51]	2016	Nursing	Non-randomized controlled study	FC: pre-session online videos, learning materials, pre-class assignments and quizzes; class time devoted to case-study and group discussion.Control: traditional lecture	Second-year and third-year Nursing students	Treatment (Second-year, n = 99)Control (Third-year, n = 52)	1: Satisfaction survey2b: Standard test (physical assessment and common symptoms assessment ) and Note writing	Control group performed better on physical assessment test (P < .05); FC improved active learning, provided more flexibility, but time-consuming.
Qiu et al[52]	2017	Stomatology	Historical	FC: pre-session online videos, learning materials; class time devoted to review.Control: traditional lecture	Medical students	Treatment ( n = 35)Control (former year, n = 35)	2b: Standard test (knowledge test and practice test)	FC group performed better on both knowledge test and practice test (P < .05)
Cao et al[53]	2016	Stomatology	Historical	FC: pre-session online videos, learning materials, group discussion and online test; Class-time included 18 min video, 13 min lecture, demonstration 8 min, and 15 min discussionControl: Class-time included traditional lecture, demonstration, and discussion.	Second-year and fourth-year Medical students	Treatment (Second-year, n = 39)Control (Fourth-year in former year, n = 35)	1: Post-intervention survey2b: Standard test (Knowledge test and OSCE)	FC group performed better on test (P < .05); FC group students satisfied with the FC method.
Yan et al[54]	2018	Biochemistry	Historical	FC: Learning group forming through WeChat, pre-session online videos, learning materials, group learning with leading questions; class time devoted to student group presentations, and discussion.Control: traditional lecture	Second-year Medical students	Treatment (n = 514)Control (former year, n = 510)	1: Post-intervention survey2b: Standard test	Control group performed better on test (P < .05); FC group students satisfied with the FC method; FC improved active learning, self-learning ability, teamwork and problem solving ability.
Zhang et al[55]	2018	Biochemistry	Randomized controlled study	FC: Learning group forming through WeChat, pre-session online videos, learning materials, case-study, pre-class assignments and tests; class time devoted to discussion.Control: traditional lecture	Second-year Medical students	Treatment (n = 49)Control (n = 48)	1: Satisfaction survey2b: Standard test	Control group performed better on test (P < .05); FC group students satisfied with the FC method; FC improved learning efficiency, self-learning ability, teamwork and problem solving ability and communication.
Wang et al[56]	2017	Obstetrics and gynaecology	Randomized controlled study	FC: pre-session online videos, learning materials; class time devoted to discussion.Control: traditional lecture	Third-year Medical students	Treatment (n = 100)Control (n = 100)	1: Self-evaluation survey2b: Standard test	FC group performed better on test (P < .05); FC group students rated higher in learning ability, learning efficiency, and learning attitude (P < .05).
Ding et al[57]	2019	Physiology	Non-randomized controlled study (selected by groups)	FC: pre-session online videos, learning materials; class time devoted to discussion.Control: traditional lecture	Second-year Medical students	Treatment (n = 124)Control (n = 122)	1: Post-intervention survey2b: Standard test	FC group performed better on test (P < .05); FC improved self-learning ability, teamwork and problem solving ability and faculty-student communication.
Li et al[58]	2017	Pathology	Non-randomized controlled study (selected by groups)	FC: Learning group forming through MB, pre-session online videos, learning materials, case-study; class time devoted to discussion.Control: traditional lecture	Third-year Medical students	Treatment (n = 49)Control (n = 51)	1: Satisfaction survey2b: Standard test	FC group performed better on test (P < .05); FC group students satisfied with the FC method
Chen et al[59]	2018	Medical Diagnostics	Randomized controlled study	FC: Learning group forming through WeChat and QQ, pre-session online videos, PPTs, learning materials, case-study; class time devoted to discussion.Control: traditional lecture	Third-year Medical students	Treatment (n = 70)Control (n = 70)	1: Self-evaluation survey2b: Test (Knowledge test and Clinical skills test)	FC group performed better on clinical skills test (P < .05); Control group students rated higher in learning ability, learning efficiency, and self-learning ability (P < .05).
Qian et al[60]	2016	Medical Diagnostics	Non-randomized controlled study (selected by groups)	FC: pre-session online videos, PPTs, learning materials, pre-class quizzes; class time devoted to review and discussion.Control: traditional lecture	Second-year Medical students	Treatment (n = 124)Control (n = 119)	1: Satisfaction survey and interview2b: Test (Knowledge test and Clinical skills test)	FC group performed better on both knowledge and clinical skills test (P < .05); FC improved self-learning ability, teamwork and problem solving ability and communication skills.
Zheng et al[61]	2019	Anatomy	Randomized controlled study	FC: pre-session online videos, readings, learning materials, group learning with case-study; class time devoted to student group presentations, and discussion.Control: traditional lecture	First-year Medical students	Treatment (n = 25)Control (n = 25)	2b: Test (Case analysis)	FC group performed better on case analysis (P < .05)
Zhang[62]	2015	Histology and embryology	Non-randomized controlled study (selected by groups)	FC: Learning group forming through WeChat and QQ, pre-session online videos, quizzes and experiment practice; class time devoted to in-class online test and discussion.Control: traditional lecture	Medical students	Treatment (n = 36)Control (n = 36)	1: Post-intervention survey2b: In-class online test	FC group performed better on test (P < .05); 80.56% FC group students satisfied with the FC method; FC improved self-learning ability, teamwork and problem solving ability and communication skills.
Ma et al[63]	2019	Pathology	Non-randomized controlled study (selected by groups)	FC: Learning group forming through WeChat, pre-session online videos, PPTs, group learning with case-study and leading questions; class time devoted to student group presentations, and discussion.Control: traditional lecture and case-study	Medical students	Treatment (n = 218)Control (n = 221)	1: Post-intervention survey2b: Standard test	There is no difference between the two groups on test (P < .05); 96.02% FC group students satisfied with the FC method.
Guan et al[64]	2018	Internal medicine	Non-randomized controlled study (with pre-test)	FC: pre-session online videos, learning materials, pre-class quizzes; class time devoted to review and discussion.Control: traditional lecture	Third-year and Fourth-year Medical students	Treatment (Third-year, n = 106)Control (Fourth- year, n = 107)	1: Post-intervention survey2b: Standard test (knowledge test and case analysis)	FC group performed better on case analysis (P < .05); 81% FC group students satisfied with the FC method.
Wang et al[65]	2018	Emergency medicine	Randomized controlled study	FC: pre-session online videos, case-study, pre-class quizzes; class time devoted to student group presentations, and discussion.Control: traditional lecture	Fourth-year Medical students	Treatment (n = 30)Control (n = 30)	1: Satisfaction survey and interview2b: Test (knowledge test and case analysis)	FC group performed better on test (P < .05); FC improved self-learning ability, effectiveness, and active learning (P < .05).
Tao et al[66]	2016	Nursing	Non-randomized controlled study (selected by groups)	FC: pre-session online videos, case-study, pre-class assignments; class time devoted to student group presentations, and discussion.Control: traditional lecture	Second-year Nursing students	Treatment (n = 32)Control (n = 30)	1: Dundee ReadyEducation Environment Measure (DREEM)2b: Standard test	FC group performed better on test (P < .05); FC group students got a higher score on environmental perception, teacher perception and learning perception (P < .05).
Guo et al[67]	2017	Biochemistry	Non-randomized controlled study (selected by groups)	FC: pre-class assignments and group learning with leading questions; class time devoted to student group presentations, and discussion.Control: traditional lecture	Second-year Medical students	Treatment (n = 204)Control (n = 184)	1: Post-intervention survey2b: Standard test	FC group performed better on case analysis (P < .05); 80% FC group students satisfied with the FC method.
Liu et al[68]	2016	Pathology	Non-randomized controlled study (selected by groups)	FC: pre-session online videos, case-study, pre-class assignments; class time devoted to student group presentations, and discussion.Control: traditional lecture	Second-year Medical students	Treatment (n = 42)Control (n = 43)	1: Satisfaction survey 2b: Self-rating of self-directed learning2b: Standard test (MCQ)	There is no difference between the 2 groups on test (P < .05); FC group students had higher self-directed learning ability (P < .05);FC improved self-learning ability, teamwork and problem solving ability and communication skills. (P < .05).
Zhang et al[69]	2018	Histology and embryology	Non-randomized controlled study (selected by groups)	FC: pre-session online videos, PPTs, group learning; class time devoted to student group presentations, and discussion.Control: traditional lecture	First-year Medical students	Treatment (n = 70)Control (n = 69)	1: Post-intervention survey2b: Standard test (MCQ, judgment questions, short answer questions)	FC group performed better on test (P < .01); FC group students satisfied with the FC method.
Guan et al[64]	2018	Internal medicine	Non-randomized controlled study (with pre-test)	FC: pre-session online videos, learning materials, pre-class quizzes; class time devoted to review and discussion.Control: traditional lecture	Third-year and Fourth-year Medical students	Treatment (Third-year, n = 106)Control (Fourth- year, n = 107)	1: Post-intervention survey2b: Standard test (knowledge test and case analysis)	FC group performed better on case analysis (P < .05); 81% FC group students satisfied with the FC method.
Wang et al[65]	2018	Emergency medicine	Randomized controlled study	FC: pre-session online videos, case-study, pre-class quizzes; class time devoted to student group presentations, and discussion.Control: traditional lecture	Fourth-year Medical students	Treatment (n = 30)Control (n = 30)	1: Satisfaction survey and interview2b: Test (knowledge test and case analysis)	FC group performed better on test (P < .05); FC improved self-learning ability, effectiveness, and active learning (P < .05).
Tao et al[66]	2016	Nursing	Non-randomized controlled study (selected by groups)	FC: pre-session online videos, case-study, pre-class assignments; class time devoted to student group presentations, and discussion.Control: traditional lecture	Second-year Nursing students	Treatment (n = 32)Control (n = 30)	1: Dundee ReadyEducation Environment Measure (DREEM)2b: Standard test	FC group performed better on test (P < .05); FC group students got a higher score on environmental perception, teacher perception and learning perception (P < .05).
Guo et al[67]	2017	Biochemistry	Non-randomized controlled study (selected by groups)	FC: pre-class assignments and group learning with leading questions; class time devoted to student group presentations, and discussion.Control: traditional lecture	Second-year Medical students	Treatment (n = 204)Control (n = 184)	1: Post-intervention survey2b: Standard test	FC group performed better on case analysis (P < .05); 80% FC group students satisfied with the FC method.
Liu et al[68]	2016	Pathology	Non-randomized controlled study (selected by groups)	FC: pre-session online videos, case-study, pre-class assignments; class time devoted to student group presentations, and discussion.Control: traditional lecture	Second-year Medical students	Treatment (n = 42)Control (n = 43)	1: Satisfaction survey 2b: Self-rating of self-directed learning2b: Standard test (MCQ)	There is no difference between the 2 groups on test (P < .05); FC group students had higher self-directed learning ability (P < .05);FC improved self-learning ability, teamwork and problem solving ability and communication skills. (P < .05).
Zhang et al[69]	2018	Histology and embryology	Non-randomized controlled study (selected by groups)	FC: pre-session online videos, PPTs, group learning; class time devoted to student group presentations, and discussion.Control: traditional lecture	First-year Medical students	Treatment (n = 70)Control (n = 69)	1: Post-intervention survey2b: Standard test (MCQ, judgment questions, short answer questions)	FC group performed better on test (P < .01); FC group students satisfied with the FC method.

Results

The scope of published research on the FC in health professions education

Since 2014, there have been a growing number of studies on the FC in health professions education in China, with only 3 articles published at that time. In 2015, 13 articles were published, followed by 36 in 2016. The number of publications peaked in 2017 and 2018, with 79 articles published each year. This rising trend may continue as there were already 25 articles published in 2019 by the time this literature search was conducted (Figure 2). Most of the articles were non-empirical research (eg, commentaries or action research).

Figure 2.

Publication trend.

Use of FC was evaluated in 17 biomedical domains across the 84 quantitative studies. There were 10 articles in biochemistry, 7 in pathology, 3 in anatomy, and 9 in internal medicine. Nursing and clinical diagnostics had 6 articles each. Radiology, stomatology, histology and embryology, preventive medicine, and surgery had 4 articles each. Anesthesiology, obstetrics and gynecology, and neurosurgery had 2 articles each. Otorhinolaryngology, dermatology, and pediatrics had 1 article each. Fourteen of the research papers were not specific to any particular medical specialty.

Our search included only the undergraduate level of health professions education. Most articles focused on students in the second and third years, and only 5 were focused specifically on students in the clerkship year (Year 4 or 5). In addition, most of the studies focused on students cohorts of less than 100.

The quality of published studies on flipped classroom in health professions education in China

Quality of the published research was evaluated based on the measures of effectiveness. Of the 59 controlled studies, 26 used random selection of participants and 33 used convenience selection.

The FC treatment consisted of pre-class activities and in-class sessions. Pre-class activities consisted of a variety of online methods: videos and other learning materials (PPTs, readings and websites information) to deliver content, and online engagement tools, such as questions, group discussion, case study, assessments and experiments. A few of the studies also provided pre-class orientation or guidelines to help students adapt to the new teaching method. Pre-class online tutorials were available in some interventions. A wide range of in-class activities were represented in the papers and included student group presentations, reviews, case analysis, and discussion.

The effects of flipped classroom

In order to evaluate the effectiveness of the FC on students’ learning, 59 studies using a controlled design were selected for further analysis. Table 2 provides a summary of each study. In the category of controlled studies, 3 distinct designs were recognized: historically controlled study (n = 3); randomized controlled study (n = 26); and non-randomized controlled study (n = 30). The historically controlled studies compared the experimental group to a control group that was offered during a different term or school year. Learners were randomly selected in the randomized controlled studies. The participants were selected by convenience in the non-randomized controlled study (eg, selected by groups). Each of these designs consisted of at least one control group (eg, traditional lecture) and one experimental group (FC). Kirkpatrick’s framework was used to analyze the results of the effectiveness of the FC reported in these 59 studies.

Perceptions of flipped classroom

A total of 53 controlled studies assessed students’ perceptions of the flipped classroom. Among them, 34 of them assessed the satisfaction of the FC group and control group, while 19 of them assessed only FC groups. Most used self-reported satisfaction or self-assessment surveys although 1 used an interview, a teaching quality assessment instrument, and the Dundee Ready Education Environment Measure. Students generally responded favorably to this type of instruction; most of them agreed that FC methodology was very helpful improving their self-learning ability, problem solving ability, teamwork, and communication skills (n = 39). Six studies indicated that students were satisfied with the engagement with faculty members and peers.[2,19,20,22,24,31] However, students who received FC instruction reported increased learning load.[2,19,23] In addition, feedback from students utilizing FC noted that the instructional method was less efficient compared with traditional lectures.[33,51]

Knowledge and skills changes with the flipped classroom curriculum

In 58 of the 59 controlled studies, changes in knowledge and skills were assessed as outcome variables. Knowledge test, skills test, and case analysis were most commonly used to document achieved course competencies in these studies (Table 2). To evaluate the effectiveness of FC teaching approach, almost all of the studies applied a statistical t-test with a 95% confidence interval. Research that examined the changes in knowledge and skills of students using FC strategy versus the traditional lecture revealed mixed results; most resulted in positive findings, 3 in negative, and 4 in negligible differences. For example, Wang et al found that students in the FC group performed significantly better on both test and dental sculpting skills (P < .05).[14] Sun et al found that students in the lecture group performed better on physical assessment test than students in the FC group (P < .05).[51] Xiao et al found no significant difference between students in the FC and lecture groups on the test (P < .05).[25] In addition, 25 of the 59 studies were carried out in the clinical learning/courses of medical school and 24 of them evaluated the effects of FC. Among the 24 research, 23 studies examined gain in knowledge and skills when using the FC compared with traditional lecture, and only one of them showed that there was no statistically significant effect between FC and lecture. The effect of the FC on learning has been evaluated across the lower levels of Kirkpatrick’s model: learner perception, changes in learner attitude and changes in learner knowledge and skills. However, among the 59 studies, there was little evidence of changes in behavior, organizational practice and patient outcome.

Discussion

Studies reflected a wide range of implementation strategies for using time before class to present information and time in class for discussion or other forms of student engagement making it difficult to assert broad inferences about education effectiveness. Despite this challenge, we identified 3 key findings in the current FC literature in China. First, since 2014 there has been an increased number of research papers on the effectiveness of the FC approach, with an increase in empiric rather than descriptive studies. In the past 2 years, the quantity of the studies on FC has increased, as well as in the number of the empiric studies. This suggests that not only the quantity of the research increased, but also the quality. This trend needs to be continued in order to develop a strong research basis for the FC method application in health professional education in China. In addition, the empirical research covered almost all the health profession specialties, which indicates that all practitioners in this area were trying to improve student learning experiences. Moreover, contrary to the earlier systematic reviews which focused mainly on research carried out in the pre-clinical learning curricula of medical school,[7] this review may provide more information of the effectiveness of FC in the clinical learning. More studies aimed at clinical years with the patient care environment are needed to explore the evidence of changes in behavior, organizational practice, and patient outcome.

Second, we found that a large proportion of Chinese students prefer the FC teaching method over traditional lectures. Previous studies in health professional education similarly found that more students favored the methodology, especially the self-paced learning before class and increased engagement in class.[3,7] However, students also reported FC required a significant time investment before class. The current body of evidence suggests that pre-class work may be excessive.[70] Faculty members may want to consider avoiding student overload - instead of containing excessive detail, pre-class work should cover key points pertaining to in-class content.[17,20,24,30,67] Better design with reduced content should be considered for the instructors to promote the learning experiences. For instance, Hew and Lo suggested that the total length of the pre-class video should not exceed 20 minutes.[3] In addition, in-class education design should be connected with pre-class assignments to promote deep learning. Furthermore, student orientation may be required to improve the FC learning experience. By explaining the rationale behind the FC, students may be able to understand the advantages of this form of learning.[71] Guidance and support with the pre-class activities may lead to better student preparation for class and increase the satisfaction rate.

Third, students learning through FC methodology found improved self-learning ability through the pre-class online videos and learning materials, as well as increased communication skills and teamwork through group learning and discussion, enhanced problem solving ability and clinical skills through in-class highly engagement, and enhanced faculty-student and peer communication through the active learning experiences. A systematic review including 29 Chinese articles in nursing education reported that self-directed learning was significantly improved with FC methods compared with traditional teaching.[72] However, few studies included self-regulated learning ability as a variable to evaluate the effectiveness of FC.[73] Future research should consider including self-regulated learning ability as a variable to test the effects of FC, as there is evidence that student preparation is a key factor of effectiveness of the flipped classroom.[8] Moreover, different media should be applied to engage students in a variety of activities.[71] In China, QQ and WeChat are popular platforms and using this kind of media may improve students’ participation and satisfaction.

Fourth, this review indicates a number of positive findings in students’ knowledge and skills from the FC approach. This result is similar to that obtained by Hew and Lo, van Alten et al., and Chen et al.[3,7,73] Several studies using a knowledge and clinical or practical skills test, reported increased academic performance with the FC methodology. However, no articles evaluated educational outcomes related to students’ higher order thinking skills, such as critical thinking or creative thinking. There was little study of the long-term improvement in learning outcomes using the FC method compared to lectures. In addition, assessment methods need to align with the desired outcomes to evaluate the effectiveness of a teaching methodology.[3,7,33] Critical thinking and problem solving ability are common desired education outcomes of techniques that incorporate active learning yet these traits are not assessed in the current literature. However, existing research primarily relied on knowledge and skills tests to evaluate the learning outcomes. Testing for superficial knowledge acquisition may not sufficient for assessing the FC approach since this teaching method emphasizes application.[3] Future research should consider the relationship between desired outcomes, instructional design, and assessment.

Furthermore, faculty development and technology team support should be implemented to assist faculty members with preparation of pre-class resources and educational design. Researchers found that quality of pre-class videos were an important factor influencing learning, student satisfaction, and completion of assignments,[12,15,36,44,68] however many faculty members found video preparation and production to be challenging and time-consuming. Duration of FC curriculum, selection of appropriate subject matter, and student learning skills should be taken into account when designing FC and other curricula.[32,52,54] Learning management systems may improve or encourage pre-class video completion rate to enhance the learning effectiveness.[17]

This systematic review has several limitations. First, the strict exclusion criteria may have led to the omission of articles. For example, articles that indicated application of the FC but were reinforced by other educational methods were omitted from this review. Second, due to the quantity of research on FC, this review only included undergraduate level of health professional education in China. Third, only 2 independent reviewers were involved in this review. More independent reviewers’ involvement may be considered in future review to improve the validity. Finally, this review only included studies published in Chinese; future reviews with both Chinese and English literature may build a stronger evidence base.

Conclusion

Our review suggests that the FC is a promising teaching method in health professional education in China which may significantly improve learning, especially at the clinical level. In addition, reducing content and enhancing student orientation and support may lead to better a FC learning experience and improved outcomes. Future studies should consider a pre-test to provide a more robust understanding of the nature of change associated with the FC methodology. Longitudinal study could be employed to assess the long-term effects of the FC strategy on student learning.[5] Future investigations should also consider evaluating the higher levels of Kirkpatrick’s outcomes.