Review of flipped learning in engineering education: Scientific mapping and research horizon.

The COVID-19 pandemic makes flipped learning more relevant to address the challenges of remote learning. Therefore, renewed attention is warranted in critically appraising the implications on which flipped learning is built. Though several studies have reviewed the flipped learning research in the past, the majority has qualitatively synthesized the flipped learning literature, thus, lacking the overall perspective provided quantitatively for appraising the existing state of affairs of flipped learning research in engineering education. This study addresses this gap by objectively mapping the conceptual, intellectual, and social structure of research development in flipped learning using a bibliometric review method. Findings reveal that flipped learning in engineering education is a relatively new field of research and in recent time it has entered into the stage of exponential growth. Findings also show the effectiveness of the flipped learning model to address the challenges of complex pedagogical applications in different fields of engineering education. This study provides a quantitative synopsis of the flipped learning literature which can be used as an anchor for future study.

Introduction

The COVID-19 pandemic highlights a unique challenge in education to address the complexities of remote learning. Educational institutions are looking for immediate solutions that can effectively deliver courses to remote learners using a cost-effective approach. This urgent situation has drawn increased attention among the educational research community to the flipped learning pedagogical approach because of its inherent nature of promoting interactive learning at low cost (Chick et al., 2020; Danjou, 2020; Fogg & Maki, 2020; Tang et al., 2020).

The notion of flipped learning is not new; it has, however, become more viable and considered more valuable in recent years due to easy access to modern technologies, internet connectivity, and up-to-date resources (L. Cheng et al., 2019; Sun et al., 2018; Zhai et al., 2017). As the name implies, flipped learning is a learner centered approach that reverses the traditional teaching and learning process (P. W. Cheng et al., 2019; L. Cheng et. al., 2019; Keengwe et al., 2014; Oncel & Kara, 2019). Through this process, teachers are required to deliver pre-recorded class lectures and course materials online before the class, so that face-to-face contact is dedicated to collaborative group discussions and extensive problem solving (Mok, 2014; Oncel & Kara, 2019; Tsai & Wu, 2020). Research shows many benefits of the flipped learning approach such as increased learning outcomes (L. Cheng et al., 2019; Jang & Kim, 2020), effective project-based teaching (P. W. Cheng et al., 2019), positive student attitude (Kang, 2015), improved teacher-student and student-student interaction and active learning (Della Ratta, 2015), meta-cognition (Van Vliet et al., 2015), problem solving (Chun & Lee, 2016), and student engagement (Jang & Kim, 2020). Though, like every other pedagogical approach, it comes with potential weaknesses such as increased class size reduces the potential benefits of the flipped method (Hotle & Garrow, 2016; Van Vliet et al., 2015), increases teachers’ workloads, and offers minimal student interaction during the pre-recorded class lectures (Acedo, 2013; Hotle & Garrow, 2016). However, over the years the popularity of flipped learning has gained momentum with researchers from different countries continually contributing to its growth and influence.

In recent years, several articles have reviewed the flipped learning literature (Bequette, 2019; Ismail, 2019; Jang & Kim, 2020; Karabulut-Ilgu et al., 2018; Lo & Hew, 2019; Lundin et al., 2018; Safapour et al., 2019; Voronina et al., 2017). Among them, Lo and Hew (2019) examined the effects of flipped education on student achievement through conducting a meta-analysis of studies occurring in engineering disciplines; Voronina et al. (2017) critically evaluated and synthetized the flipped learning literature to examine the existing state of flipped education; Safapour et al. (2019) recognized flipped classroom as a non-traditional learning method and explored the different benefits of flipped learning; (Lundin et al., 2018) examined knowledge contributions within this field to relate them to education technology available for higher level study; Ismail (2019) reviewed comprehensively the current state of flipped learning intertwined with education technology in an engineering department; Jang and Kim (2020) focused on quantifying the effects of flipped classrooms in higher education keeping in mind the cognitive, affective, and interpersonal outcomes.

Despite revealing the efficacy of the flipped learning approach, previous review papers suffer from some critical limitations. First, the majority of the studies have qualitatively reviewed the flipped learning literature, thus, the sample selection procedure is subjectively biased. Second, these papers lack the objectivity to envisage the existing state of affairs of flipped learning approaches; this process is facilitated by mapping the scientific knowledge area based on scientometric data to critically review and characterize the research ideas and the subsequent challenges. In addition, the extant literature differs markedly from the intended aims of the current review paper which is entirely focused on the status quo of flipped learning occurring in engineering education. Although, Karabulut-Ilgu et al. (2018) systematically reviewed the use of the flipped learning method in engineering education, it provides only a qualitative synthesis of previous studies. Furthermore, Tsai and Wu (2020) recently performed a bibliometric analysis but it is lacking a review of the status of flipped learning research in engineering education. Therefore, no existing research studies have objectively studied the developments in flipped learning research within engineering education. Considering these inadequacies of previous review papers, this study intends to provide a comprehensive quantitative overview of the flipped learning literature in engineering education using the bibliometric review technique. In so doing, this study attempts to address the following research questions:

How has the focus of the flipped learning literature in engineering education evolved over time since its widespread acceptance in 2013?

What documents, authors, institutions, sources, and countries are the most eminent and influential in the flipped learning literature?

How have the publications and citation trends changed over time?

What is the core conceptual, social, and intellectual clusters nested in flipped learning and how have different authors and countries contributed to those clusters?

Methodology and data

This paper applies the bibliometric review technique for examining the status quo of flipped learning applications in engineering education to ascertain the key research streams and trends in recent time. A number of review techniques exist including the structured review, the review for model/framework development, meta-analysis, a theoretical review, hybrid-future research, framework based, and the systematic review (Snyder, 2019). While many approaches are potentially available, in this study the bibliometric approach has been applied to objectively envisage and map the flipped learning research in engineering education. The application of the bibliometric technique has the potential to induce richer insights due to its inherent design of quantitatively synthesizing the research topic of any discipline based on citation mapping (Zupic & Čater, 2015).

To date, this study is the first bibliometric analysis on flipped learning in engineering education. The database drawn upon in this analysis is the Web of Sciences (WoS) core collection. This source has become the most prominent platform for bibliometric analyses as it includes the most impactful publications required for science mapping and identification of future research directions (Maditati et al., 2018; Piñeiro-Chousa et al., 2020).

This study ultimately selected 106 articles published between year 2013 and 2020 from the entire WoS database. In the search criteria, “All years (1900-2020)” and “All documents” were enabled so that no relevant documents were excluded in the search results. The schematic flow diagram illustrated in Fig. 1 presents the criteria for article selection, procedural steps, and the review strategy of this study. This figure is adopted from Khan et al. (2020).

Fig. 1

Article selection process and review methodology

In stage 1, the search included all possible keywords to reduce the chance of missing any important document. To elicit the more relevant documents, Boolean operators were utilized in the search strings- (“flipped” OR “inverted”) AND (“learning” OR “classroom” OR “teaching” OR “instruction”) AND (“engineering education” OR “engineering”). The search returned 559 documents. In stage 2, conference proceedings were excluded, and documents selected were restricted to articles, review articles, book chapters and early access, a process which returned 194 documents. Restricting the search to English language only returned 174 documents. Further research discipline refinement by excluding chemistry, physics, mathematics, science technology other topics, environmental science ecology, social science and other topics, agriculture, business economics, information science and library science, reduced the corpus to 155 documents. In stage 3, titles and abstracts screening was undertaken to exclude papers with no scientific relevance to flipped learning in engineering education. This abstract and title screening returned a final 106 documents.

The repository of 106 articles produced in this manner formed the basis for the bibliometric analysis. This review was carried out in relation to five different components to encompass the comprehensive mapping of flipped learning in engineering education: i) article production and citations, ii) impactful authors, institutions, countries and sources, iii)) conceptual structure of scientific fields, iv) intellectual structure of the connected research themes v) social structure of the scientific collaboration. For each component, the number of publications, citations, co-citations, keywords, the most productive authors, sources, countries, and organizations were identified. To transact this process, the study employed the visualization of similarity (VoS) viewer software package, the bibliometrix package in R and the HistCite software program to map the flipped learning research.

Bibliometric review and discussions

In this section a deeper insight is provided into the documents under review, that is, how they quantitatively and qualitatively contribute to the domain of flipped learning. Scientific mapping of the bibliometric information of these documents has been achieved on the conceptual, intellectual, and social structure levels. A total of 106 documents, authored by 272 authors, have been selected for analysis. These documents are comprised of 97 journal articles, 5 review articles, and 4 early access journal articles. These documents, published between 2013 and 2020, were located in 51 sources. A total of 909 citations of these documents were counted as of the end of July 2020 within the web of science database of which 141 references were self-cited. The average citation per document is 8.57 with a h-index of 13. Furthermore, there exists 16 single authored documents and on the average 2.8 co-authors have contributed per document with a collaboration index of 2.84.

Article productions and citations

The major impact of the scientific articles within the engineering education domain began in 2013. However, the annual scientific production has increased rapidly from the year 2017, in fact, 82% of the scientific papers were produced after 2017. This rapid growth in scientific papers was connected to an increased number of citations (Fig. 2) such that, 86.67% of the total citations were made after the year 2017 with an annual growth rate of 32.05%.

Fig. 2

Annual Scientific Productions Vs Citations report within Web of Science Core Collection, as of July 2020

Though there has been a sudden reduction in both the published research and citation of articles in 2020, this is due to the fact that the current study has only considered studies up to July 2020. The trend lines indicate that both the productions and citations will likely trend higher by the end of 2020.

Although, article citations rate increased after the year 2017, interestingly, the most cited articles were published before 2015 (Fig. 3). In fact, only 10 articles were published between 2013 and 2015 and they account for 54% of the total citations (47.7% in 2013, 2.8% in 2014, and 3.5% in 2015). This finding indicates that most of the later publications may have less influence on the flipped learning research community. It is possible the early published articles are ground-breaking; that is, the ‘pioneers’ have set the parameters which have become the reference points for authors subsequently. However, practically and logistically, it is understandable that the most recent articles (e.g., published after 2018) have been cited on fewer occasions as some time is required for their full impact among the research community to be registered.

Fig. 3

Average Article Citation Per Year

The impact of authors, institutions, countries, and sources

Figure 4 illustrates the productivity patterns of authors in the domain of flipped learning. This pattern has been drawn using Lotka’s law (Lotka, 1926). It explains the frequency of the publications by the authors in a specific field of study in a given time frame. The authors’ productivity pattern in Fig. 4 illustrates an inverse relationship between the percentage of authors and the number of documents created. It thus means that in regard to flipped learning the percentage of authors decreases with the increase of document numbers.

Fig. 4

Number of documents written by the authors (in %)

Figure 4 also shows that a major portion of the authors have written only one document compared to what is expected from the ideal pattern of article writing, that is, 91.5% of authors wrote a single article focusing on a flipped learning domain which is much higher than the ideal case scenario. In contrast, 7.7% authors wrote two, with only 0.7% authors contributing three documents to the flipped learning research, much lower than the ideal publication pattern as perceived in other high interest areas. This implies two things: first, many authors with one publication did not consider flipped learning as their main research area; second, in the future, according to Lotka’s law, some of these authors may contribute further to the flipped learning literature.

As flipped learning has received attention only recently, its impact in the domain of engineering education might be considered to be in its early days. Consequently, the maximum number of articles written by any author is recorded as only 3 (Table 1). Only two authors, Aliye Karabulut-Ilgu and Lesya Hassall have contributed to that extent. Numerically, they are the most statistically impactful authors till July 2020. Collectively, both have produced 5 articles, one of which they co-authored. Interestingly, these articles were published recently between 2017 and 2019. Three of the articles mainly focus on students’ perspectives such as student perception, student team dynamics and student collaboration. One of the articles focuses on the teachers’ perspective and the other article has undertaken a systematic review of flipped learning in the engineering education. The review article by Karabulut-Ilgu et al. (2018) quickly became influential among the flipped learning research community. In just two and half years, this article has been cited 82 times, the second highest cited article in the flipped learning research field.

Table 1

The 20 most impactful authors, institutions, and countries in flipped learning

R	Author	NP	GC	SC	H	C/A	Institutions	NP	GC	SC	H	C/A	Country	NP	GC	SC	H	C/A
1	Karabulut-Ilgu A	3	103	2	3	34.3	Iowa State University	5	107	2	3	21.4	USA	34	606	35	10	17.8
2	Hassall L	3	21	0	2	7	State University System of Florida	5	40	1	2	8	Spain	17	40	4	4	2.4
3	Cook KE	2	337	1	2	168.5	Central Michigan University	3	43	1	3	14.3	Taiwan	6	85	1	3	14.2
4	Shuman TR	2	337	1	2	168.5	Universidad Politecnica De Madrid	3	15	1	2	5	Australia	6	23	0	2	3.8
5	Mason GS	2	337	1	2	168.5	Universidad Carlos III De Madrid	3	6	1	2	2	Turkey	6	9	0	2	1.5
6	Cherrez NJ	2	99	1	2	49.5	University of South Florida	3	4	1	1	1.3	China	5	10	0	2	2
7	Drake E	2	32	1	2	16	Pennsylvania Commonwealth System of Higher Education Pcshe	3	3	1	1	1	South Korea	3	12	0	2	4
8	Yelamarthi K	2	32	1	2	16	University of Pittsburgh	3	3	0	1	1	Brazil	3	9	0	2	3
9	Diaz I	2	9	0	1	4.5	Polytechnic University of Catalonia	3	2	0	1	0.67	Malaysia	3	5	0	2	1.7
10	Castedo R	2	6	1	2	3	Tecnologico De Monterrey	3	0	0	0	0	Mexico	3	0	0	0	0
11	Chiquito M	2	6	1	2	3	Seattle University	2	337	1	2	168.5	New Zealand	2	16	0	1	8
12	Alario-Hoyos C	2	6	0	2	3	National Changhua University of Education	2	54	0	1	27	Russia	2	14	0	2	7
13	Estevez-Ayres I	2	6	0	2	3	University of Florida	2	36	0	2	18	Germany	2	10	0	1	5
14	Lopez LM	2	6	1	2	3	University of Nebraska Lincoln	2	5	0	2	2.5	France	2	2	0	1	1
15	Medina-Merodio JA	2	5	1	2	2.5	Universidad De Alcala	2	5	1	2	2.5	India	2	2	0	1	1
16	Baughman J	2	4	0	1	2	University of Nebraska System	2	5	0	2	2.5	Norway	2	2	0	1	1
17	Kloos CD	2	4	0	1	2	Near East University	2	3	0	1	1.5	Ethiopia	2	0	0	0	0
18	Munoz-Merino PR	2	4	0	1	2	Karadeniz Teknik University	2	2	0	1	1	Serbia	2	0	0	0	0
19	Clark RM	2	2	0	1	1	Monash University	2	2	0	1	1	Sweden	1	26	0	1	26
20	Fredriksen H	2	2	0	1	1	University of Agder	2	2	0	1	1	Qatar	1	13	0	1	13

Compiled from the citation report generated by the Web of Science web-application. Ranked by number of publications (NP) and then by total global citations (GC) and then by citation per article (C/A). SC = self-citation; H = h-index

As illustrated in Table 1, Cook, Shauman and Mason became the most influential authors to date as their articles have produced 337 citation - the highest number of citations in the flipped learning literature as of July 2020. They also collaborated as co-authors producing two articles. The first article they wrote, Mason et al. (2013), is the topmost cited article with 334 citations. This highly cited article focused on three important aspects of flipped learning- learning content, student performance, and student perception. It also investigated the effectiveness of the flipped classroom in these three areas compared to the traditional classroom.

Table 1 further shows that the most impactful institution in flipped learning research is the Iowa State University with the highest number of publications (5) and citations (107). The next two most influential institutions are the State University System of Florida (5 articles) and Central Michigan University (3 articles). Both universities’ publications have attracted more than 40 citations. Interestingly, the top 3 most influential universities are from the USA. When considering citations only, Seattle University, which is also located in the USA, rises to the top position with 337 citations. This data conforms with other findings in which the USA appears to be the most influential country in producing 34 publications and 606 citations. The third most citated institution is the National Changhua University of Education from Taiwan with 54 citations. This also aligns with the impact data ranked by country, based on which Taiwan achieves the third most impactful position in terms of article production and citations. Taiwan also has the second most citation per article ration (14.2) after the USA (17.8). When considering both the number of publications and citations, Spain achieved the position of the second most influential country.

Table 1 also shows a congruent relationship between the most influential author (Karabulut-Ilgu), institution (Iowa State University) and country (USA). USA also achieves the top position when considering the highest citation by authors (Cook, Shuman, & Mason), highest citation by the institution (Seattle University) and highest citations per article (17.8). In summary, USA is the most influential and dominant country in flipped learning research within the engineering education domain.

Table 2 shows that the most influential article is produced by Mason et al. (2013) and published in IEEE Transactions on Education. This article also achieves top position as the most locally and globally cited document in the flipped learning domain. In fact, among the top six locally cited (LC) and globally cited (GC) articles, IEEE Transactions on Education published three of them. Interestingly, a recent article by Karabulut-Ilgu et al. (2018), published in the British Journal of Educational Technology ranks second in both LC and GC just two years after publication. This article has undertaken a systematic review of the articles published between 2000 and May 2015, documenting the design and development process of flipped learning in engineering education. Altogether, the three most productive sources in the flipped learning domain are – the International Journal of Engineering Pedagogy (4 articles) followed by IEEE Transactions on Education (3 Articles) and the European Journal of Engineering Education (3 Articles).

Table 2

20 Most locally cited articles and sources

R	Articles	LC	GC	Sources	NP	GC
1	Mason et al. (2013), IEEE Transactions on Education	41	334	Computer Applications in Engineering Education	15	80
2	Karabulut-Ilgu et al. (2018), British Journal of Educational Technology	12	82	International Journal of Engineering Education	15	50
3	Chao et al. (2015), Computer Applications in Engineering Education	11	54	IEEE Transactions on Education	6	386
4	Yelamarthi and Drake (2015), IEEE Transactions on Education	11	25	European Journal of Engineering Education	6	36
5	Chiang and Wang (2015), International Journal of Engineering Education	8	17	Education for Chemical Engineers	5	62
6	Kim et al. (2014), IEEE Transactions on Education	6	24	International Journal of Engineering Pedagogy	3	17
7	Lucke et al. (2017), European Journal of Engineering Education	4	15	International Journal of Electrical Engineering Education	3	7
8	Battaglia and Kaya (2015), International Journal of Engineering Education	4	11	International Journal of Interactive Design and Manufacturing	3	0
9	Hotle and Garrow (2016), Journal of Professional Issues in Engineering Education and Practice	3	15	International Journal of Educational Technology in Higher Education	2	27
10	Yan et al. (2018), International Journal of Engineering Education	3	5	Australasian Journal of Educational Technology	2	18
11	Ravishankar et al. (2018), European Journal of Engineering Education	3	5	Journal of Professional Issues in Engineering Education and Practice	2	18
12	Li and Huang (2017), Education for Chemical Engineers	2	23	International Journal of Mathematical Education in Science and Technology	2	15
13	Baytiyeh and Naja (2017), European Journal of Engineering Education	2	11	On the Horizon	2	5
14	Yelamarthi et al. (2016), Journal of Information Technology Education-Innovations in Practice	2	7	Journal of Engineering Education	2	4
15	Kanelopoulos et al. (2017), International Journal of Engineering Pedagogy	2	7	Education Sciences	2	1
16	Karabulut-Ilgu et al. (2018), Journal of Educational Computing Research	2	4	British Journal of Educational Technology	1	82
17	Oddsson and Unnthorsson (2017), International Journal of Engineering Education	2	2	Attic-Revista D Innovacio Educativa	1	4
18	Kakosimos (2015), Education for Chemical Engineers	1	13	Architectural Engineering and Design Management	1	3
19	Braun et al. (2014), International Journal of Engineering Pedagogy	1	10	Computer Science Education	1	1
20	Hoole et al. (2015), Computer Applications in Engineering Education	1	3	Computers & Chemical Engineering	1	1

Sources are ranked by local citations (LC) and then by Total global citations (GC)

In contrast, when considering the total publications produced from the 106 articles, the most impactful journals achieving top position is held co-jointly by Computer Applications in Engineering Education and the International Journal of Engineering Education. Each of them produced 15 articles. Second position also is achieved co-jointly by IEEE Transactions on Education (6) and the European Journal of Engineering Education (6). However, when considering the most impactful sources in terms of total citations, IEEE Transactions on Education has become the most cited journal (386) followed by the British Journal of Educational Technology (82) which sits in 16th position in terms of total publications.

When looking into the reference lists of the 106 articles, it is found that a total of 3715 references are cited from 2170 sources. Among these sources, Table 3 shows that the most cited source is Computers & Education. A total of 141 articles have been cited from Computers & Education. The second most cited journal is IEEE Transactions on Education with 110 cited articles followed by the Journal of Engineering Education with 102 cited articles. These three are the most prolific journals in assisting authors’ writing the selected 106 articles. Interestingly, the ‘thesis’ was the 6th most important contributing sources for writing these articles. This clearly indicates that a trend towards flipped learning research has been initiated by academic institutions recently with many postgraduate research students investigating the impact of flipped learning in the engineering education.

Table 3

10 most locally cited sources from the reference list (within the collection)

R	Sources	Total Article Cited
1	Computers & Education	141
2	IEEE Transactions on Education	110
3	Journal of Engineering Education	102
4	Internet and Higher Education	67
5	International Journal of Engineering Education	57
6	Thesis	56
7	British Journal of Educational Technology	42
8	Computers in Human Behavior	42
9	Educational Technology and Society	37
10	Computer Applications in Engineering Education	35

TAC Total Article Cited, TAC/D Total Article Cited per document

A three-field Plot (Fig. 5) of country, keyword, and sources of the published articles visualized by the Sankey diagram (Riehmann et al., 2005) was created to depict the proportion of key research themes derived from each country and the association of the themes with the sources. The keywords used in the publications generally locates the research themes. The chart visually emphasizes the flows of key research ideas within selected systems, e. g., countries and sources. The color of the bars does not have any internal meaning; however, the width of the colored bars indicates the percentage of total keywords produced by the country and sources. The width of the chart lines is shown proportionally to the flow quantity.

Fig. 5

Three field plots of Country-Keywords-Source based on the Sankey diagram

The key ideas related to flipped learning, as depicted in the chart, can be categorized into three main themes- flipped classroom, learning, and higher education. First, the ‘flipped classroom’ emerged as the most dominant keyword alongside other relevant keywords like ‘flipped’, ‘classroom’ and ‘inverted classroom’. This clearly shows the researchers have primarily focused on understanding the purposes and nature of the flipped classroom. This is understandable as it is a relatively new domain in educational research. Second, researchers have focused on ‘learning’ itself within the flipped classroom settings. Researchers have blended online and face to face learning to create the flipped context which demands active participation, problem solving skills, collaboration to investigate the outcomes of this type of learning. These together identify the nature of flipped learning in which students are required to engage actively in the learning process. Third, the researchers have investigated flipped learning in the context of higher education, that is, inclusive engineering education.

Figure 5 further shows that the USA and Spain researchers are dominating in flipped learning research followed by China and Brazil. We also see that sources such as the International Journal of Engineering Education, Computer Applications in Engineering education and IEEE Transections on Education, compared to other sources, have disproportionately contributed to the corpus of research, covering several themes intrinsic to flipped learning.

Conceptual structure

Keyword co-occurrence analysis becomes a key approach in bibliometrics to demonstrate the conceptual structure of the connected research themes (Cheng et al., 2018; Radhakrishnan et al., 2017). The following conceptual structure in Fig. 6, produced by VOSviewer, represents the most closely connected research keywords in flipped learning. The minimum co-occurrence of a keyword set by the researcher is 3. Out of 328 keywords, 20 keywords met this threshold. The color of the nodes determines different clusters formed by the keywords. The size of the label and the node of any keyword is determined by the number of occurrences of the keywords. Lines between the nodes represent the relationships between the keywords. The distance between two nodes indicates the strength of association between the keywords. Thus, the closer the two nodes are, the greater the co-occurrence existing between the keywords.

Fig. 6

Co-occurrence of the 20 most representative author keywords

Figure 6 shows six clusters of author keywords in which three of them (green blue and red colored clusters) include most of the keywords. Essentially these three clusters indicate the dominant research topic in the flipped learning domain. As illustrated the green node has the highest occurring keyword i.e. ‘flipped classroom’ (59) followed by ‘active learning’ (22). Further this green cluster explains a learning environment that provides personalized and a flexible learning path sometimes incorporated with ‘adaptive technology’ and ‘gamification’. Students are required to adapt their learning preferences to a non-traditional gamified flipped learning environment though the small size of the nodes of ‘gamification’ and ‘adaptive technology’ indicate that little research has been transacted in this area of flipped learning. Researchers basically include educational gaming elements into flipped classroom as a method to increase participation and interest by providing a game such as challenges in the learning process. The blue colored cluster is led by ‘flipped learning’ (14) and ‘engineering education’ (10). This blue cluster frames educational innovation as the core research theme. It informs us about interactive teaching and the development of strategic and the autonomous learning environment for students to promote problem-solving skills within engineering education. In the next dominating cluster (red) ‘blended learning’ (12) is the highest occurring keyword. The red cluster illustrates the strong relationships between student-centered learning and the e-learning environment in which collaborative and problem-based learning components are employed to facilitate the flipped learning approach

The other three clusters have connections with keywords such as ‘learning analytics’ with a strong relation to ‘higher education’, and ‘meta-analysis’ of flipped learning research. In relation to Fig. 6, Table 4 further shows the total links and total link strength of the co-occurrence of the 20 most representative author keywords. These 20 keywords create 57 links and produce total link strength of 117 (data extracted from VOSviewer). The sum of individual link strength connected to the node determines the total link strength of that node. As depicted, only the top six keywords from the Table 4 have a total link strength more than 10.

Table 4

Co-occurrence and link strength of Author keywords

Keywords	Occurrences	Links	Total links strength
Flipped classroom	59	18	61
Active learning	22	14	36
Flipped learning	14	10	18
Blended learning	12	9	22
Engineering education	10	8	16
Higher education	7	6	11
Inverted classroom	6	4	8
Learning Analytics	5	6	10
Collaborative learning	5	4	6
Problem-based learning	4	4	5
Meta-analysis	3	4	6
Student-centered learning	3	4	5

Clusters of keywords and their interconnections are further used for mapping the research concepts of flipped learning in engineering education. A two-dimensional strategic diagram has been ploted (Fig. 7) to extract four kinds of research themes (Cahlik, 2000; Cobo et al., 2011; He, 1999).

Fig. 7

Visualising and mapping the themes of flipped learning through the strategic diagram

The strategic diagram illustrates the clusters of keywords in four quadrants. According to the bibliometric approach proposed by Cobo et al. (2011), the keywords in the upper-right quadrant such as the flipped classroom, teaching, e-learning, undergraduate students, and so forth are known as motor-themes which are well developed and important for the structuring of the flipped learning research domain. The keywords in the lower-right quadrant are important for a research field and known as basic themes as they require further development. Themes in the upper-left quadrant make an insignificant contribution to the field. These themes are very specialized and often considered to be isolated to the field. The keywords in the lower-left quadrant such as gamification, cloud computing, project-based learning etc. are less researched and represent either emerging or disappearing themes (Cobo et al., 2011).

Intellectual structure

The application of citation mapping for unveiling underlying intellectual property of any group of documents in a bibliometric study is common (Fetscherin & Heinrich, 2015; Janik et al., 2020; Maditati et al., 2018).The citation map (Fig. 8) was created based on the 30 most locally cited articles in flipped learning. Most locally cited articles are considered to be the key literature pertaining to a topic (Cooper, 1988). Y-axis of Fig. 8 represents the year of publication; the number of articles is published in parenthesis. The citation nodes are represented as circles with an article ID record being provided for each of the 106 articles in the repository. The arrow indicates the source of cited articles. This citation mapping encapsulates the citation network of the most influential articles relating to flipped learning that have been carried out since 2013.

Fig. 8

Citation mapping of the 30 most locally cited articles

(Node. Articles): 5. Mason et al. (2013); 6. Braun et al. (2014); 7. G. J. Kim et al. (2014); 8. Chiang and Wang (2015); 9. Battaglia and Kaya (2015); 10. Cruzado and Roman (2015); 11. Hoole et al. (2015); 12. Chao et al. (2015); 13. Kakosimos (2015); 14. Yelamarthi and Drake (2015); 16. Yelamarthi et al. (2016); 18. Petrillo (2016); 22. Harrison et al. (2017); 23. Baytiyeh and Naja (2017); 24. Cook et al. (2017); 28. Oddsson and Unnthorsson (2017); 29. Kanelopoulos et al. (2017); 33. Lucke et al. (2017); 34. Li and Huang (2017); 38. Voronina et al. (2017); 39. Turner and Webster (2017); 41. Ravishankar et al. (2018); 45. Yan et al. (2018); 48. Munir et al. (2018); 52. Y. Kim and Ahn (2018); 54. Rodriguez et al. (2018); 73. Castedo et al. (2019); 77. Kaw et al. (2019); 79. Lin (2019); 84. L. Cheng et al. (2019). [Note: HistCite failed to show Hotle & Garrow, 2016; Karabulut-Ilgu et al., 2018 in the citation mapping for an unknown reason. The respective position of these two articles are 9th and 16th in the most locally cited article ranking (see Table 2)].

The citation mapping suggests several research streams of flipped learning for which the most locally cited articles are contributing. These research streams are presented in four different clusters in Fig. 8. They are: 1) the effectiveness of the flipped learning model; 2) the evaluation of the flipped learning model; 3) student perception of the flipped learning model; and 4) the application of the flipped learning model to solve problems in teaching-learning.

Model effectiveness

The ‘model effectiveness’ cluster has only one article written by Mason et al. (2013). However, considering the influence of this article among other most locally cited articles, it deserves a cluster on its own. Total local and global citations produced by Mason et al. (2013) are 41 and 334 respectively. Almost all other articles have cited Mason et al. (2013) and therefore created an umbrella shape network (Fig. 8). The network formation indicates that the work of Mason et al. (2013) has contributed to most areas relevant to flipped learning. This article primarily investigates the effectiveness of the flipped learning model on covering the course content, performance of students, and their satisfaction with the flipped classroom. Mason et al. (2013) argued that compared to a traditional classroom context, the flipped learning method helped teachers to cover more subject material and to support better students’ performance. Mason et al. (2013) also found that students were satisfied more with the flipped classroom than they were with the traditional learning arrangement.

However, not every article among the most locally cited documents support the findings of Mason et al. (2013). A study conducted by Cruzado and Roman (2015) revealed that the flipped learning approach was neither favorable nor unfavorable in terms of students’ performance. Nonetheless, this study measured positive student attitudes toward the flipped learning approach. Even more remarkable, one study demonstrated the negative impact of flipped learning on students’ performance compared to traditional learning instructions. In this study, Harrison et al. (2017) utilized the flipped learning in an undergraduate fluid mechanics course and employed active learning components by reducing seat time for the traditional classroom component. They found a marginally significant negative impact both in exam scores and on homework performances. As the study revealed, this negative impact might be due to the poor video usage employed in the flipped learning approach.

Model evaluation

The articles in this cluster further extend the work of Mason et al. (2013) by evaluating the flipped learning model in different dimensions related to teaching-learning. A number of researchers have used the flipped learning method to investigate how this method can improve student learning and active engagement with the courses, for example, student retention (G. J. Kim et al., 2014), reducing failure rates (Petrillo, 2016), promoting student success through doing exercises (Oddsson & Unnthorsson, 2017), engaging students without overloading them with class hours (Ravishankar et al., 2018) and so forth. Hoole et al. (2015) incorporate several active learning strategies and found increased student engagement and interest. Another key dimension of student learning is self-efficacy investigated by Yan et al. (2018). They found that flipped learning improved learning motivation by promoting intrinsic goal orientation and control of beliefs which eventually strengthened student self-efficacy for learning. In promoting active student learning, researchers have attempted to evaluate the usefulness of flipped learning. Yelamarthi et al. (2016) suggest that the active and collaborative learning component in the flipped learning method improved student learning. More recently, Castedo et al. (2019) complemented these findings by suggesting that highly engaged and active students can utilize the flipped learning method best in their learning.

Researchers have also evaluated the flipped learning model from the teachers’ perspective. Chiang and Wang (2015) found that teachers can gain access to students easily and can organize their course contents well in pursuing this approach. Teachers also utilized the method well to tackle the challenges of time constraints in covering a large volume of course material (Battaglia & Kaya, 2015; Hoole et al., 2015). Teachers too can address budget constraints in their teaching with low-cost technology aids to enhance student learning (Yelamarthi & Drake, 2015). Further, Yelamarthi et al. (2016) argued that flipped teaching can be cost effective and a good alternative to teacher delivery of lectures, reducing the need for traditional classroom spaces.

Additionally, two articles evaluate the usefulness of the flipped learning method by doing extensive literature reviews. Karabulut-Ilgu et al. (2018) qualitatively synthesized the quantitative and qualitative research in the flipped learning research extant in engineering education. They found that research in this field concentrated on authenticating the design and development process of the flipped learning model as well as determining the usefulness of it. L. Cheng et al. (2019) undertook a literature review to evaluate the overall effect of the flipped classroom on student learning outcomes. This meta-analysis compared flipped classrooms with the traditional classrooms, and they found the results were significantly in favor of the flipped classroom approach.

Model perception

The articles in this group emphasize students’ perception and attitudes towards the flipped learning method. For example, Braun et al. (2014) reported in their study the positive attitude of students; specifically, the possibility to learn at one’s own pace in a flipped learning environment has been applauded by students. Voronina et al. (2017) also reported positive students’ attitudes towards flipped learning. In particular, this learning approach had a positive effect on the transfer of learning- a key finding related to students’ learning attitudes and motivation that were reflected in students’ perception of the flipped learning approach (Chao et al., 2015; Kakosimos, 2015). Additionally, students believed that their experience within the flipped learning environment was the cause of the changes in their learning perspectives which specifically improved their critical and analytical skills (Munir et al., 2018).

Model application

Research studies that go beyond the understanding of the effectiveness and usefulness of the flipped learning approach are accumulated under this cluster. Some articles in this cluster emphasize more the solving of problems associated with engineering education using the flipped learning approach. For example, Cook et al. (2017) used the flipped learning method to solve authentic engineering problems (AEPs) in a heat transfer course in order to improve students’ problem solving skills. AEPs are authentic problems developed by practitioners to replicate the unstructured, open-ended problems encountered by engineering industries. Similarly, Lucke et al. (2017) used the flipped learning approach to realize the effectiveness of student-response system (SRSs) in terms of student engagement, motivation and cognition.

Further, several other studies tried to integrate the flipped learning approach with innovative technological tools in a pedagogical framework to address different issues within engineering education. For example, a study developed an instructional design framework recommending wikis as the hub of content delivery (Kanelopoulos et al., 2017). This study used collaborative problem-solving as the key method to employ in classrooms as a part of a flipped learning approach. Furthermore, researchers have used flipped learning in a number of contexts including: an inquiry learning approach (Kim & Ahn, 2018), in a peer instruction and gamification learning structure (Rodriguez et al., 2018), in an adaptive lessons platform (Kaw et al., 2019), in a smart learning diagnosis system (Lin, 2019) and so on to enable the solving of engineering problems. The flipped learning model has been employed in lab classes too. For example, Li and Huang (2017) utilized the flipped learning approach in mathematical model simulations using the application of MATLAB/Simulink in an engineering lab course.

Social structure

The network and collaboration between authors are playing a key role in understanding the research direction in the various research fields. Often this collaboration creates research hubs that drive the growth and development of that research field. This network collaboration can be visualized by mapping the co-authorship occurrences among the published documents. The co-authorship network shows the scholarly associations among the research communities in different institutions and countries (Donthu et al., 2020).

Figure 9 portrays the mapping of such authorship co-occurrences between the countries in flipped learning research. This social network signifies the extent of collaboration between the countries (Liao et al., 2018). The shape of the circles signifies the number of publications led by the country. Two primary collaborative networks emerged in flipped learning research led by USA (red cluster) and Australia (green cluster). These are the two most influential countries in flipped learning driving collaborative research in this field. Interestingly, both the collaborating networks started to build their connectivity and momentum recently, more specifically after the year 2017.

Fig. 9

Mapping and visualizing the authorship co-occurrences between the countries

Further, to understand the collaboration strength between the countries, Table 5 shows the strength of the links between the countries. The thickness of lines and distance between circles indicate the collaboration strength. The total strength of a country’s links is determined by the number of documents published by the authors by which they represent two or more different countries.

Table 5

Co-authorship network

Network 1				Network 2
Country	Links	Total links strength	Documents	Country	Links	Total links strength	Documents
USA	4	6	34	Australia	4	5	6
China	3	5	5	Malaysia	4	5	3
Germany	3	3	2	Serbia	4	4	2
England	3	3	1	Scotland	3	3	1
Papua New Gini	1	1	1	Wales	2	2	1
				Slovenia	1	1	1

Implications and limitations

This study has a number of implications for researchers and academicians. It serves as a directory for future researchers and academicians to identify the most eminent authors, institutions, sources, and countries involved in flipped learning research. This study also provides a quantitative synopsis and qualitative synthesis of the flipped learning literature which can be used as an anchor for future study.

However, this study suffers from some limitations such as only the web of science database was sourced. Also, book chapters, books and conference papers were excluded from this study. Future research endeavors might include other prominent databases such as Google scholar and Scopus to adopt different review approaches to provide a more comprehensive quantitative and qualitative synthesis of the research front in this field.

Conclusions

This study uses bibliometric analysis techniques to provide a quantitative synopsis and qualitative synthesis of the flipped learning literature from 2013 to 2020. The findings reveal that in terms of productivity, no single author, institution, or sources dominated the literature, although in terms of influence, Seattle University and IEEE Transactions on Education are found to be the most relevant institution and source, respectively. The flipped learning literature has drawn attention from researchers of various countries, particularly those from technologically advanced countries, notably the USA, Spain, Taiwan, and Australia. The USA appears to be the most eminent country in this field publishing the greatest number of documents on flipped learning; as well, many of the most eminent authors are also from the USA. The social structure analysis explores the two most influential countries in collaborative research, the USA and Australia. The publication trends indicate that the flipped learning literature has entered into the stage of exponential growth, although the citation ratio still remains low; one reason behind this may be that for recently published documents, it requires some time to create impact. Only a few documents have received more than 100 citations and very few keywords have been used more than 10 times. The keyword co-occurrence network analysis identifies different clusters of flipped learning research in which ‘blended learning’, ‘higher education’, and ‘engineering education’ reveal an immense opportunity for future research work to be conducted. Finally, the intellectual structure identifies four different clusters inherent in the flipped learning model that uncover its effectiveness to address the challenges of complex pedagogical applications in different emerging fields of engineering education.