Do project-based learning, hands-on activities, and flipped teaching enhance student's learning of introductory theoretical computing classes?

The need for computer science (CS) education, especially computer network education, is increasing. However, the challenges of teaching students with diverse backgrounds and engaging them in hands-on activities to apply theories into practices exist in CS education. The study addressed the challenges by using project-based learning (PBL) and flipped teaching approaches to cover both theoretical and hands-on learning aspects in CS education. This study aims to demonstrate the design and development journey of a CS course and examine whether using PBL, hands-on activities, and flipped teaching approaches improves students' learning. The design-based research study was conducted in an undergraduate CS course from 2014 to 2020 at a midwestern university. The design and development trajectory in the six years were described. The descriptive statistics were used to analyze the trends of the course evaluation results, and ANOVA were conducted to examine whether the evaluation differs from each semester. The results indicated that using PBL, hands-on activities, and flipped teaching increased students' learning motivation and their perceptions of their learning. Combining PBL and flipped teaching appropriately can enhance students' learning motivation and perceived learning in CS education, but further research is needed to examine how each individual intervention influence students' learning motivation and learning outcomes.

Introduction

Computer Science related occupations are projected to be one of the fastest-growing jobs through 2029, and the need for a workforce with computing skills is increasing (U.S. Bureau of Labor Statistics, 2021). However, it is challenging to teach introductory computing classes (Kay et al., 2000). In particular, for the courses that expect learners to acquire both theoretical and practical knowledge and skills, one of the primary challenges is to elevate junior and sophomore students’ interest in learning the theoretical foundation of computing subjects.

In the past three decades, the evolution in the field of Computer Networks has transformed almost every aspect of our lives. Internet, also called ‘Network of Networks,’ is the largest system ever developed, with hundreds of millions of connected end-systems (e.g., computers, servers, smartphones, etc.), communication links, and switches (routers and linked layer switches). Each of these components runs different protocols (set of rules in the form of software) that interact with its counterparts in other components.

Computer networks, an introductory class of 4-years Information Technology (IT), Computer Science (CS), and Cybersecurity curriculum, allows junior or sophomore students to develop an understanding of networking technology; gain factual, conceptual, and applied knowledge regarding network communications, security, infrastructure components, and design issues; and prepare them for the advance classes focusing on cybersecurity operations (e.g., detection and prevention of cyber-attacks) and systems and IT infrastructure operations (server, workstation, virtualization, and cloud computing environments).

Given that the Internet is so large, and it involves complex concepts of digital communication, network software, reliability, communication among different components of the Internet and humans, network security principles, it is difficult to teach all these challenging theoretical and applied concepts to sophomores’ students coming from different background, particularly when there is only one course in the whole undergraduate curriculum. With evolving Accreditation Board for Engineering and Technology (ABET) guidelines and vast development of the computer science field, it is becoming challenging to offer more than one class in a single sub-specialty (e.g., Database, Computer Networks, Artificial Intelligence) of information technology (IT) and CS 4-year degree. Therefore, it is important to redesign classes that offer both intermediate and advanced level knowledge but also both theoretical and hands-on knowledge to make students capable of meeting market needs and fulfilling ABET requirements at the same time.

The present article presents how the researchers designed, developed, and revised the Introduction to Computer Networks course at the Midwest University in the United States to meet ABET and industry requirements, integrating different pieces of knowledge and skills to develop a deeper understanding of computer networks, and to offer a good balance between theory and hands-on activities. This class used the Internet as a vehicle to explain the theoretical and practical aspects of computer networking.

The course was taught by multiple instructors between 2014 and 2017, who found it difficult to receive a satisfactory end-of-the-semester course evaluation. Several challenges existed due to the following reasons: (1) The students of the course, from CS and IT programs, had diverse technical backgrounds; (2) to cover the vast field of computer networks, there is only one four-credit hours class in four years curriculum, and it is offered in the second year of the program; (3) to cover the theoretical foundations and principles of the dynamic field of computer networking, practical insights, and offering hands-on (lab) activities from both aspects of IT (i.e., designing, configuring, and troubleshooting networks) and CS (i.e., writing network software). Furthermore, it is challenging for the students to understand how different protocols working on different layers of the TCP/IP layering model work in an incoherent fashion; (4) to cover the lab component, which was required to be offered in both face-to-face and online settings.

Facing the challenges mentioned above, the first author of this study, along with the undergraduate committee, redefined the course objectives and then introduced various interventions, including hands-on activities, PBL, and flipped teaching, to improve the course contents and teaching approaches from winter 2015 to fall 2020. The purpose of this study is to examine whether PBL, hands-on activities, and flipped teaching can improve course quality to meet the course objectives and to demonstrate the design and development of the course. This article addresses the following research questions:

Does hands-on practice using tools such as Wireshark and Packet Tracer improve students’ perceptions of whether the course learning objectives were met?

Do PBL, hands-on activities, and flipped teaching increase the motivation and interest level in the subject?

Do PBL, hands-on activities, and flipped teaching increase student’s perception of class time and instructional aids usage?

Do students consider the value of textbooks and recitation contribute to learning?

What are the overall students’ learning experiences and perceptions of the instructor?

Literature review

Computer science education in higher education

Computer and information technology-related occupations are one of the fastest-growing jobs through 2029 (U.S. Bureau of Labor Statistics, 2021). The employment of computer and information research scientists is predicted to increase 15% from 2019 to 2029, which is much faster than the average for all occupations (U.S. Bureau of Labor Statistics, 2021). Moreover, data indicated that almost 75% of STEM occupations require computing knowledge and skills (U.S. Bureau of Labor Statistics, 2014). In addition, a majority of professional occupations expect employees to have at least basic computation competency (Hoffmann, 2012). Among the occupations in the computer and information technology field, 69% require employees to have at least a bachelor’s degree, with a typical entry-level with a master’s degree (U.S. Bureau of Labor Statistics, 2021). Among the majors in computer and information technology, 98% of students are in computer science (CS) (U.S. Bureau of Labor Statistics, 2021). Therefore, undergraduate CS education is critical to prepare employees for the job market.

CS education covers diverse domains, including programming, databases, software design and development, and computer networks, etc. (Stephenson et al., 2005). Given the increasing importance of computer networks, the foundation of computer network education is vital. However, research on educating computer networks in higher education is rarely based on our systematic literature search and review. Thus, it is critical to explore effective strategies for computer networks education in higher education.

The survey results of the employers indicated that employees are expected to have critical thinking and problem-solving skills, collaboration skills, communication skills, and project-management skills (Mills & Treagust, 2003). However, in CS education in higher education, the courses primarily taught isolated knowledge and skills without synthesis and integration (Jaime et al., 2016). Integrating different pieces of knowledge and skills to develop a deeper understanding is vital (Rice & Shannon, 2016). The courses that only focus on conceptual and theoretical concepts is not sufficient for students to face the requirement of the job market. Instructors should create an engaging, authentic, and collaborative learning environment for students (Rice & Shannon, 2016). Project-based learning (PBL) is one approach to address the challenges and educate students with the competence needed in the job market.

Project-based learning

PBL is an approach to motivate learners and improve their problem-solving abilities (Velez & Power, 2020) using authentic projects (Al-Balushi & Al-Aamri, 2014; Dilekli, 2020). PBL, a learner-centered instructional approach, has three characteristics: (1) context-specific, (2) active involvement, and (3) social interaction (Cocco, 2006). The projects are usually complex tasks with challenging problems and questions that encourage students to actively investigate, design, and make decisions to solve problems (Doppelt, 2003). Generally, in PBL, learners collaboratively work in a group to solve open-ended problems, including practicing research inquiry and creating a final product (Helle et al., 2006; Usher & Barak, 2018). The final product could be presented as videos, graphics, reports, models, applications, and other artifacts (Holubova, 2008). In the PBL process, learners develop their understanding of the knowledge or develop new attitudes (Holubova, 2008).

Prior research has examined the effectiveness of using PBL in higher education (Kokotsaki et al., 2016). For instance, researchers have utilized an integrated PBL method in engineering majors (Hassan et al., 2008). Fernandes et al. (2014) found that using PBL can engage students’ learning. In contrast, other researchers identified the relationships between self-directed learning readiness and PBL outcomes (Stewart, 2007). Havenga (2015) found that the implementation of PBL may enhance the development of their self-directedness. In addition, the research found that PBL promoted collaboration (McManus & Costello, 2019) and critical thinking skills (Loes & Pascarella, 2017).

Particularly, in CS education, the implementation of PBL in the courses is increasing (Joorabchi et al., 2013; Kizaki et al., 2014). One of the reasons for adopting PBL is that it enables students to practice and implement the principles and procedures in a learning environment that is similar to an authentic job (Yadav & Xiahou, 2010). Balyk et al. (2021) implemented PBL in computer modeling courses and found that PBL stimulated students’ interest in learning and deepened students’ knowledge. And Jaime et al. (2016) adopted PBL, spiral learning, and peer assessment in computer science education and found that spiral PBL was effective in improving students’ learning. In addition, Francese et al. (2015) used the PBL approach to teach CS students mobile application development and provide opportunities for students to work on hands-on activities and found that students were satisfied with the teaching approach. However, research on educating computer networks in higher education, especially using PBL, is lacking.

Flipped teaching

Flipped teaching is a new instructional model in which students watch video lectures at home and participate in active activities such as discussion, problem-solving, hands-on activities, etc., in the classroom (Bergmann & Sams, 2012; Sohrabi & Iraj, 2016). In flipped teaching, students must be responsible for their learning and control their own learning (Lai & Hwang, 2016). Flipped teaching has been used widely across different disciplines worldwide (Hao, 2016).

Researchers have explored the benefits and challenges of using flipped teaching approach. For instance, Missildine et al. (2013) found that flipped teaching can enhance students’ learning achievement and satisfaction. Moreover, the flipped teaching approach was found to be able to engage students’ learning (Tomas et al., 2019). In addition, O’Flaherty and Phillips (2015) found that flipped teaching is more economical compared to traditional classroom teaching. On the contrary, flipped teaching also faced challenges such as time-consuming to redesign the course (Schlairet et al., 2014), lack of self-regulation skills of learners (Sun et al., 2017), and possible failure due to not watching the lecture before the class (Lai & Hwang, 2016). For example, Towey (2015) reflected on his failure of implementing PBL in CS courses and found that he should think further on how to facilitate and evaluate students’ group projects. This study explored using flipped teaching approach to engage students and enhance their learning. Similarly, Maher et al. (2015) presented how they utilized flipped approach in undergraduate CS courses and found that students were positive towards flipped teaching in general. The studies mentioned above combined both face-to-face sessions with online sessions in the flipped teaching approach.

Flipped teaching approach could also be used in purely online courses, in which the face-to-face session is replaced with the online synchronous meeting. In an online flipped class, students are asked to watch pre-recorded video lectures before the synchronous meeting. In synchronous meetings, instructors provide students opportunities to participate in class activities and conduct hands-on tasks. However, few studies explore how to integrate flipped teaching into online education. This study will present how we use flipped teaching in online education during the COVID-19 pandemic.

Method

This study used an educational design-based research (DBR) approach, which aimed to improve teaching practices through multiple iterations and to build design principles and theories (Anderson & Shattuck, 2012; McKenney & Reeves, 2018). The different phases of DBR (Reeves, 2006) help researchers and practitioners explore and analyze the initial problems, revise the instructions, test, and refine the design solutions in the process (Herrington et al., 2007). In DBR, the design process does not follow a prescriptive procedure; rather occurs based on the results of the retrospective analysis of the emerging situations (Mor & Winters, 2007). This retrospective analysis guides each design iteration, including the design, implication, analysis, and redesign phases (Design-Based Research Collective, 2003). The reasons for using the DBR approach in the present study included: (1) examining the effectiveness of using PBL and flipped teaching, (2) tracking the iterative design, development, implementation, and evaluation process, and (3) building design principles and theories for CS courses. 

In DBR studies, researchers can use diverse methods to obtain data (Anderson & Shattuck, 2012; Bradley & Reinking, 2011). The data sources could be both qualitative and quantitative data. In this study, course evaluation and instructor’s reflection were used as data sources.

Context and participants

As stated above, the study context was an undergraduate course, CS 2220 Introduction to Computer Networks, a required course for undergraduate sophomore students in CS and IT majors, at a Midwest University from winter 2014 to fall 2020. Six iterations of the course have been conducted. One of the authors of this study taught this course for nine semesters, i.e., winter 2014, winter 2015, fall 2015, winter 2017, fall 2017, fall 2018, winter 2019, fall 2019, and fall 2020. The number of students in the course each semester varies from 28 to 39 (see Table 1).

Table 1

Course evaluation response rates

Semesters	Total enrolled	No. of students who participated in the evaluation	Response rate
Winter 2014	39	22	56.41%
Winter 2015	32	21	65.63%
Fall 2015	28	11	39.29%
Winter 2017	34	23	67.65%
Fall 2017	35	18	51.43%
Fall 2018	33	16	48.48%
Winter 2019	30	16	53.33%
Fall 2019	36	22	61.11%
Fall 2020	35	16	45.71%

To meet ABET requirements, most universities include the following topics in this course: a review of the foundational protocols used on the Internet and modern enterprise networks, services, and topologies, information security, high-speed LAN/WAN topologies, client–server technology, security vulnerabilities, and network troubleshooting.

To accomplish the course objectives (see Table 2) and above listed topics, in this course, students learn the development and structure of protocols of Transmission Control Protocol/Internet Protocol (TCP/IP) stack, their role in data communications across the Internet, and integration within network-based applications. To gain hands-on skills, students engage in hands-on activities aimed at protocol analysis, network troubleshooting, and protocol-related security concerns. Particularly for offering hands-on activities, the following two tools were employed: Wireshark protocol analyzer and Cisco packet tracer. Wireshark protocol analyzer is introduced to assist students in understanding individual protocol structures and how various protocols work together to transmit or receive the packets in devices and troubleshooting. Cisco packet tracer tool is employed to teach configuration of different devices, explain connectivity among devices to make local and wide area networks, and troubleshooting. The following section details the course design and development process.

Table 2

Course objectives

No	Items
1	Describe network protocol stacks such as application, transport, network, link, and physical layers (1,2,6a&b)
2	Describe local area networks and wireless and mobile networks (2,6b)
3	Conduct experiments by using a network protocol analyzer such as Wireshark (1)
4	Explain major design and performance issues in computer networks (1, 6a, 6b)
5	Describe basic concepts of network security(1,6b)

Course design and development

Figure 1 shows the entire process and the various interventions introduced from Winter 2015 to Fall 2020. The instructor taught the course for the first time in the winter of 2014. At that time, the course learning objectives of the courses were: (1) To understand the theory and technology that is foundational for computer networks; (2) to consider the practices required to build and manage computer networks; (3) to provide hands-on experience with computer networks; (4) to explore the best practices in the field. Based on students’ course evaluation results, the instructors realized that the course learning objectives were not measurable and clear to students. According to Bloom’s Taxonomy (Armstrong, 2010), the learning objectives can be categorized into six levels from low to high: (1) remember, (2) understand, (3) apply, (4) analyze, (5) evaluate, and (6) create. Given that the course was in the introduction level, the first author of this study and his colleagues refined the course learning objectives into the first three levels in winter 2015. The revised five-course learning objectives were described using measurable verbs so that it is clear to both instructors and students. The five learning objectives are: (1) Describe network protocol stack such as application, transport, network, link, and physical layers; (2) describe local area networks and wireless and mobile networks; (3) conduct experiments by using a network protocol analyzer such as Wireshark; (4) explain major design and performance issues in a computer network; and (5) describe basic concepts of network security.

Fig. 1

Interventions introduced over time (winter 2015-fall 2020)

After adopting the new course objectives, new challenges emerged. The instructors found that students had difficulty implementing theories learned from the course into practice. In the winter and fall of 2017, instead of using the traditional lecture-based approach, the instructor introduced hands-on activities such as Wireshark labs, one Packet Tracer lab, and an open-ended PBL approach were implemented in the course. The Wireshark labs aimed to develop students’ understanding of Internet protocol stack and different protocols at each layer, interaction among different protocols/layers and design issues of protocols. The Packet Tracer lab aimed to teach students how different entities of the network interact, how to configure different devices such as computers/Mobile phones, routers/switches/Firewalls to have secure connectivity, and how to troubleshoot to understand the issues hampering the secure connectivity. Last, the open-ended PBL was optional for students who would like to work on an individual project to apply the knowledge to practice. The course evaluation results indicated that students’ perceptions regarding meeting the course learning objectives increased a bit.

To further examine whether the hands-on labs helped students’ learning, the instructor excluded the open-ended PBL and included more Packet Tracer labs in fall 2018. In addition, the instructor created instructional video recordings for both the Wireshark labs and Packet Tracer labs so that students could watch the videos when students worked on lab activities after class (see Table 3). The course evaluation results indicated that the evaluation scores on learning objectives increased significantly in fall 2018. As such, the instructor decided to keep the current instructional approach and kept revising the labs and videos of the labs in the winter and fall of 2019.

Table 3

Preparation of materials from fall 2018 to fall 2020

Semester	Lab Documents	Lab Videos	Recitation	Project-based learning	Lecture Videos	Assignments and Quizzes
Fall 2018	Formation of 6 new Packet Tracer Labs and 2 Wireshark Labs modules, Major Revision of 4 Wireshark Labs modules	Videos of 6 new Packet Tracer Labs + 6 Wireshark Labs	None	No [Open-ended Extra-credit Project only]	None	Six New assignments were prepared. Also, Quizzes were revised
Winter and Fall 2019	A new Network management lab was formed	None	None	PBL was removed	None	Assignments were revised to incorporate questions that require students to read a few sections and thus improving the value of textbook contributions
Fall 2020	The network management lab was revised. 2 Wireshark labs were improved. 1 additional wireless network (video still pending) configuration using Packet Tracer was introduced	1 new lab (along with a video) was introduced to explain how to use the lab infrastructure via VPN during the pandemic Additionally, 3 new animations were introduced to improve understanding of TCP reliability	For the first project, recitation-based learning was introduced	The final Exam was replaced with a hybrid structured and open-ended two research projects	26 short lecture videos + 26 Full-length videos were introduced to offer fully online, inverted online class offering	New Quizzes were made Assignments were significantly revised

In fall 2020, the course was shifted to an online format due to the COVID-19 pandemic. The final structure of the class offered in fall 2020 is detailed in Table 4. The course was completely redesigned by converting the pure lecture-based delivery to a highly engaging and interactive integrated lecture/lab. The integrated lecture/lab covers both theoretical and practical aspects that could now be used for both face-to-face and online delivery. The class held synchronous meetings twice every week using Zoom, a virtual conference tool. Following goals/evaluation metrics were defined to improve this class: (1) To deepen students’ interest in the subject of computer networking; motivating them to do their best work; (2) to assist and facilitate them in learning difficult materials and problem solutions by explaining and clarifying them through short videos; (3) introduction of animations to explain difficult concepts; (4) offering hands-on activities by making six Wireshark and seven Packet Tracer labs to improve overall learning objectives; (5) hybrid structured and open-ended project-based learning to achieve surface, deep, and transfer learning; (6) designing activities to keep students more engaged and better utilization of class time by offering flip teaching approach; and (7) introducing recitation component for improving learning objectives.

Table 4

The final structure of the class offered in fall 2020 (fully online) with all proposed interventions

Class Meetings	Lectures & Labs	Assignments	Recitation &Quizzes	Labs	Material prepared and used for intervention
Meeting 1	Course Introduction & Chapter 1, Protocol, Physical layer			Demo: Introduction to Wireshark and Packet tracer	Demo notes/slides, Lecture Video
Meeting 2	Chapter 1, packet & circuit switching	Assignment 1		Lab 01: Making a simple network and configuring a Router in Packet Tracer	Lab Document, Lab + Lecture Video
Meeting 3	Chapter 1, delay loss and throughput in packet switching		Recitation -Project 1	Lab 2: Introduction to Wireshark: Capturing packets and performing basic analysis	Lab Document, Lab + Lecture Videos, Project Tutorial, Recitation
Meeting 4	Chapter 2, Application Layer		Recitation -Project 1		Project Demo + Project Tutorial + Lecture/Project Videos, Recitation plan
Meeting 5	Chapter 2, Application Layer	Assignment 2	Online Quiz 1	Lab 3: Setting Up DHCP and DNS using Packet Tracer	Lab Document, Lab + Lecture Videos, Quiz
Meeting 6	Chapter 2, Application Layer, web and HTTP			Lab 4: Basic Analysis of HTTP packets Using Wireshark	Lab Document, Lab + Lecture Videos
Meeting 7	Chapter 2, Application Layer, Socket programming		Recitation – Project 1		Project Demo + Project Tutorial + Lecture/Project Videos
Meeting 8	Chapter 3, Transport Layer			Lab 5: Dissecting TCP Segments using Wireshark	Lab Document, Lab + Lecture Videos
Meeting 9	Chapter 3, Transport Layer	Assignment 3	Online Quiz 2		Lab + Lecture Videos, Quiz
Meeting 10	Chapter 3, Transport Layer			Lab 06: TCP/IP network administration/Configuration Commands in Windows and Linux	Lab + Lecture Videos
Meeting 11	Chapter 3, Transport Layer				Lecture Video + TCP animation
Meeting 12	Chapter 3, Transport Layer				Lecture Video + TCP animation
Meeting 13	Chapter 4, Network Layer: Data Plan	Assignment 4	Online Quiz 3	Lab 7: Configuring Routers using Command Line Interface in Packet Tracer	Lab Document, Lab + Lecture Videos, Quiz
Meeting 14			Presentation – Project 1		Recitation plan
Meeting 15	Chapter 4, Network Layer: Data Plan			Lab 08: IP Wireshark Lab	Lab Document, Lab + Lecture Videos
Meeting 16	Chapter 4, Network Layer: Data Plan				Lecture Video
Meeting 17	Chapter 4, Network Layer: Data Plan			Lab 9: Connecting Two remote LANS to make Wide Area Network in packet tracer	Lab Document, Lab + Lecture Videos
Meeting 18	Chapter 6, Link Layer and LAN				Lecture Video
Meeting 19	Chapter 6, Link Layer and LAN	Assignment 5	Online Quiz 4		Quiz + assignment revision, Lab + Lecture Videos
Meeting 20	Chapter 6, Link Layer and LAN			Lab 10: Analyzing Ethernet and ARP Frames using Wireshark	Lab Document, Lab + Lecture Videos
Meeting 21	Chapter 7, wireless and Mobile networks	Assignment 6			assignment revision, Lab + Lecture Videos
Meeting 22	Chapter 7, wireless and Mobile networks			Lab 11: Designing a Secure Wireless LAN in Packet Tracer	Lab Document, Lab + Lecture Videos
Meeting 23	Chapter 7, wireless and Mobile networks				Lecture Video
Meeting 24	Chapter 8, Network security		Online Quiz 5		Lecture Video
Meeting 25	Chapter 8, Network security			Lab 12: Integrity and Authentication Verification	Lab Document, Lab + Lecture Videos
Meeting 26	Chapter 8, Network security				Lecture Video
Meeting 27			Project 2 Presentations		Recitation plan
Meeting 28			Project 2 Presentations		Recitation plan

A data-driven approach was taken to continuously improve the desired outcomes by offering various interventions in fall 2020. The interventions introduced were as follow: a) Video-based learning of lab material, b) flipped teaching approach where the students watched a video of lecture offline before discussing complete lecture, exercises, and projects in class, c) hybrid of structured (tutorial-based) and open-ended PBL was offered to achieve surface, deep, and transfer learning. More specifically, structured tutorial-based project 1 (along with the code of different modules and the prepared data) was offered. In project 1, each group was offered to pick either one research or IT-based applied project. In the second project, students applied the skills learned in structured-based project 1, hands-on labs, and lab material to real-world problems. For the second project, students worked with the instructor to define the scope of the project from the given 29 topics (related to course objectives). Scrum methodology was introduced to execute the project in a group of 4–5 students. Additionally, the recitation component was introduced using structured tutorial-based project 1 (the code of different modules and the data were prepared).

Project 1 aimed to achieve Surface learning which allowed students to define and label ideas and use skills. In the second project, students worked without following structured tutorials and half-prepared solutions to achieve deep learning and transfer learning. Deep learning allowed students to relate ideas and connect skills and transfer learning, which allowed them to apply ideas and skills to different situations. Table 4 shows all proposed activities and interventions introduced in Fall 2020 (fully online). Apart from the initially proposed activities, an online lab component was offered (via VPN) to ensure that students could complete the lab component without staying in the physical Lab room, which was greatly appreciated by the students.

Data collection

To examine the effectiveness of the course design, this study used the instructor’s reflection on teaching and students’ course evaluation of course learning objectives and general experience through the six years in winter and fall semesters. The course evaluation used five Likert scales from one to five to represent to what extent the learning objectives were met in this course and their perceived learning experience in a course (evaluation items Q1-Q17) and three open-ended questions (see Appendix 1). The 17 items cover a variety of aspects, including course materials, engagement, difficulty level, facilitation, utilization of class time, interaction in class, etc.

Data analysis

This study adopted descriptive analysis and an ANOVA to compare whether the adjacent semester’s course objective evaluation results significantly differ. The statistical analysis was conducted using R, a language and environment for statistical computing and graphics. The qualitative data from open-ended questions were analyzed using thematical analysis (Braun & Clarke, 2006; Braun et al., 2014), a qualitative data analysis approach where the researchers read through data and then identified common themes across the dataset.

Results

Research question (RQ) #1 Does hands-on practice using tools such as Wireshark and Packet Tracer and PBL improve students’ perceptions of “whether the course learning objectives were met”?

First, the ANOVA results show that there are significant differences among different semesters (Fall 2020, Fall2019, Winter 2019, Fall 2018, Fall 2017, Winter 2017, and Fall 2015). In addition, between winter 2017 and fall 2015 showed that the mean of the overall evaluation of course learning objectives increased by 0.54. As mentioned before, in winter 2017, the instructor introduced an open-ended PBL approach and hands-on activities such as Wireshark labs. This indicated that using PBL and hands-on activities can have a positive influence on students’ course learning objective evaluation (Table 5).

Table 5

ANOVA analysis of the learning objectives items (Fall 2015-Fall 2020 semesters)

ANOVA
Source of Variation	SS	df	MS	F	P-value	F crit
Between Groups	1.441208	6	0.240201	4.460335	0.002748	2.445259
Within Groups	1.507878	28	0.053853
Total	2.949086	34

Second, after video-based labs were introduced, the evaluation scores in fall 2018 increased (0.34) significantly compared to that in fall 2017 (see Fig. 2). It might be because the instructor refined the Wireshark labs and introduced six new Packet Tracer labs. This resulted in an increase in applied learning objectives, which is objective 3, “conduct experiments by using a network protocol analyzer such as Wireshark.” In addition, the instructor made video recordings for the labs so that students could review them while they worked on their lab homework. This indicated that using hands-on activities along with videos on these activities had a positive impact on students’ course learning objective evaluation.

Fig. 2

Increase in the average of course objectives

Third, the analysis results indicated that students’ evaluation of course objectives in fall 2020 increased significantly (0.34) compared to fall 2019. From the fall of 2018 to the fall of 2019 semesters, the PBL approach was not used. But, in the fall of 2020, the instructor introduced structured PBL, open-ended PBL, and flipped teaching approach in the online course. The average course objective evaluation result was 4.22 out of 5. This indicated that students, in general, thought the course objectives were met in the course. Further analysis between course learning objective evaluation of fall 2018 to fall 2020 shows that revisions of the labs in fall 2020 have particularly helped to improve learning objective 3.

Overall, the average score of Q1-Q17 of the end semester evaluation also improved (see Fig. 3). In winter 2014, the average evaluation score was three out of five. By fall 2020, the average evaluation score increased to 4.1. One student from fall 2020 emphasized the importance of PBL in the open-ended questions, “the projects helped way more and learned more than just taking a test.” Another student commented, “I really appreciate the focus to be more on implementation and labs rather than rigorous testing.” In conclusion, our analysis shows that the introduction of Wireshark and Packet tracer labs along with PBL helped to integrate different pieces of conceptual and hands-on knowledge and skills to develop a deeper understanding of computer network for students.

Fig. 3

Increase in the average of course evaluation Q1-Q17

RQ #2 Do PBL, hands-on activities, and flipped teaching increase the motivation and interest level in the subject?

Our analysis of data shows that PBL, hands-on labs, and flipped teaching enhanced students’ motivation and interest levels over time, with 2.6 and 2.5 respectively out of five in winter 2014 and 4.4 in motivation, and 3.8 regarding interest level in Fall 2020 (see Fig. 4). Particularly, the motivation of students to do their best work significantly increased in fall 2020 due to flipped teaching and PBL. In conclusion, PBL, hands-on activities, and flipped teaching increase the motivation and interest level in the subject.

Fig. 4

The impact of interventions introduced in terms of increase in the motivation and interest level in the subject

RQ #3 Do PBL, hands-on activities, and flipped teaching increase student’s perception of class time and instructional aids usage?

The scores of students' evaluation of effective utilization of class time and instructional aids increased (Fig. 5). It is quite apparent from the results of Fig. 5 that PBL flipped teaching approach offering, the use of animations, and videos of labs and lectures have improved students' perception of the use of instructional aids and utilization of class timings. For example, course evaluation on items “utilization of class time” and “effectiveness of instructional aids” increased 20% from winter 2014 to fall 2020.

Fig. 5

Student’s perception of class time usage and instructional aids over time

RQ #4 Do students’ perceptions of the value of textbooks and recitation contribute to learning?

It is very challenging to encourage students to read the textbook in Computer Network courses. Therefore, in assignments, questions were made that required students read and understand certain sections. As shown in Fig. 6, overall, the value of textbook contributions in fall 2020 has increased. Moreover, in fall 2020, the recitation component was introduced with PBL. Figure 7 shows that the recitation component was evaluated much higher than in previous semesters.

Fig. 6

Improvement in value of textbook contribution due to revision of homework assignments

Fig. 7

Improvement in the value of recitation

RQ #5 What are the overall students’ learning experiences and perceptions of the instructor?

One of the important measures of students’ satisfaction is their evaluation of the overall rating of this course as a learning experience. Figure 8 shows that students' learning significantly improved in fall 2020 due to all proposed interventions compared to previous offerings. In addition, the overall rating of the instructor’s teaching also increased from winter 2014 to fall 2020. Moreover, the figure also demonstrates that overall rating as a teacher and overall rating as a learning experience has a strong correlation.

Fig. 8

Overall rating of course as a learning experience and its correlation with an overall rating as a teacher

Lastly, it is important to note that although in fall 2020, the class was offered online, students rated the lab facility as adequate. This is mainly because the instructors worked with the University Technology Services (UTS) department to make the infrastructure available online (via VPN). Students have not only used this infrastructure online for projects but also for assignments. One student from fall 2020 mentioned about the instructor “An amazing professor who deserves unending praise for his care over the students. He made sure every student understood everything throughout the course and provided recordings of the lectures for review.” Overall, students were satisfied with the course and instructor.

Discussion and conclusions

This study indicated that using PBL, labs for hands-on activities, flipped teaching approach, and video recordings of the labs and lectures have resulted in significantly increased scores at the end of the course evaluation by the students in terms of (1) overall course objective evaluation, (2) overall Q1-Q17 at the end of the course evaluation, (3) overall rating as a teacher, (4) overall course learning experience, (5) utilization of class time, (6) usage of instructional aids and (7) increase in the motivation and interest level in the subject. The findings of this study support the findings from prior studies that PBL can engage students (e.g., Fernandes et al., 2014), motivate students’ learning (e.g., Hassan et al., 2008), and deepen students’ knowledge understanding (e.g., Balyk et al., 2020). In addition, the present study findings also confirm with the study findings of Francese et al. (2015) that using the PBL approach to teach CS students and enhances students’ satisfaction in the course. Despite that prior research has indicated the importance of using PBL to engage CS learners and improve their skills to solve problems (e.g., Ali, 2005; Rice & Shannon, 2016), they primarily focused on general undergraduate courses or graduate level courses. In particular, this study offers an effective approach to teaching junior and sophomore level classes.

Moreover, flipped teaching approach adopted in the fall of 2020 also indicated that it might play an important role in enhancing students’ positive attitude towards the course and the teaching approach. As found in the study of Maher et al. (2015), students held positive attitudes towards flipped teaching. Flipped teaching creates an active learning environment, which is critical for undergraduate CS education (Ali, 2005). Ali advocated using an active learning approach such as hands-on lab activities, projects, and multimedia materials for CS education. In flipped teaching, after watching videos before class, students can engage in more active learning activities during the synchronous meeting. Based on the researchers’ conversations with students, students appreciated being able to watch the videos based on their own schedule before class. However, challenges also exist in flipped teaching approach, such as failure to watch the video lectures before synchronous class meetings. Thus, further research needed to be explored regarding the strategies to encourage students to watch videos before the synchronous meeting sessions.

The introduction of the interventions, such as labs and videos, in fall 2018 has resulted in better evaluation; however, in winter 2019 and fall 2019, the evaluation score got decreased. One of the possible reasons is that the instructor had the cognitive pressure of running multiple research projects and preparation of the dossier for tenure review. In summer 2020, all the proposed activities were revised, and the remaining were completed, which resulted in substantial improvement in the overall end-of-course evaluation in fall 2020. It is important to note that the success of interventions introduced in the fall of 2018 in the proposed CSI 2470 class has motivated the instructor to introduce similar interventions for information security class (CSI 4480), particularly hybrid structured and open-ended PBL. In winter 2021, the average of course objectives increased to 4.5 for CS 4480, which was never achieved since 2014 for CS 4480.

The instructor has also observed that in fall 2020, the average grade of the class was A- compared to B or B- in previous offerings. Through course evaluation, qualitative data analysis, and the informal conversation between the instructor and the students, the researchers found that PBL and hands-on activities are much more effective for students learning compared to quizzes and Mid-term and Final exams. The instructor has also noticed that the value of the recitation component also improves the overall learning, at least for introductory classes. Keeping students motivated by offering feedback, relating the material to cutting-edge research, and designing projects that have relevance in the industry increases students’ interest in the course.

Finally, the instructor created an online lab environment so that students could work on hands-on activities during the pandemic. The access to online labs in the fall of 2020 enabled learners to integrate theoretical knowledge with hands-on practices online. This also could possibly be used in online computer networks programs post the pandemic era and scale up online teaching.

Limitations and future research

Two limitations exist in this study. First, the research was conducted in only one CS course in the U.S. The generalization of the present study findings to the other context should be cautious. Future research could implement PBL and flip teaching in different courses in CS majors or regions. Second, this study examined the improvement of the course design and development using the end of the course evaluation of students and instructor retrospection. Future research could include more data sources such as grades and semi-structured interviews with students to get an in-depth understanding of students’ experiences to triangulate data. Third, this study did not specifically examine the possible confounding variables, such as the increase in the instructors’ experience through the years. Future studies could take confounding variables into consideration.

Conclusion

The design and development of introductory classes in IT and CS disciplines, such as computer networks, for undergraduate students, is a step-by-step, iterative process. In this study, we presented how we designed, developed, and revised the course in the past six years. We demonstrated the specific decisions that we made each year. Particularly, we emphasized how we used PBL, flipped teaching approach to teach CS sophomore students to cover various theoretical and hands-on concepts. Despite studies on the use of PBL and flipped approach that existed before, the studies that focus on using both PBL and flipped teaching in online CS education are rare. We believe that the findings of this study could provide implications for courses with comparable populations. This project is the initial stage of the study of using PBL and flipped teaching for hands-on computer networks courses. Future researchers could examine in-depth how and when to use PBL and flipped teaching approaches in similar or different undergraduate computer science courses. More confounding variables, such as teachers’ experience, students’ background, etc., could be considered in related studies.

The design and development process in this DBR study is an ongoing process. The authors of this study are continually revising and improving this course by further validating the influence of these approaches in this paper. In addition, the authors plan to explore innovative approaches to the course to personalize and improve students’ learning in CS courses.