The impact of emergency remote teaching on a blended engineering course: perspectives and implications for the future.

Higher education institutions globally were forced to transition to remote teaching and learning when the Covid-19 pandemic impacted the world in 2020. The rushed, unplanned nature of the transition led to the approach being labeled Emergency Remote Teaching (ERT). This paper evaluates the impact of ERT on a blended course in engineering using a descriptive case study approach applying the Context, Input, Process and Product (CIPP) evaluation model. The context analysis highlighted the need for consistent training in the use of technology, technical support for stakeholders, greater access to the Internet and timely, targeted communication. Students appreciated the convenience of online classes and accessibility to recorded lectures and labs allowing them to review at their own pace. There was a perception that the new learning environment placed some students at a disadvantage. These findings suggested a need to ensure deliberate planning for online learning from the start and attention to building a community of learners. Findings from the study can contribute to a university's exploration of the academic enterprise. These findings can also help identify mitigating factors for effective online learning. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s43545-021-00172-z.

Introduction

Higher education institutions throughout the world were forced to transition to remote teaching and learning when the Covid-19 pandemic impacted the world in 2020. Some institutions had been involved in the delivery of online learning before the impact of Covid-19 but traditional face–to-face (FTF) methods of teaching and learning was the main approach by the majority. Institutions struggled to continue to provide instruction with the unexpected shift to online learning which became necessary. The change was hailed by some as a positive one which could transform higher education (Govindarajan and Srivastava 2020) and by others as a temporary action to cope with a crisis. The rushed, unplanned nature of the transition led to the approach being labeled Emergency Remote Teaching (ERT) rather than online learning (Bates 2020; Hodges et al. 2020). Like other university lecturers, teaching staff at the University of the West Indies (UWI), St Augustine Campus engaged in Emergency Remote Teaching to continue instruction to their students. On this campus, an engineering lecturer had been teaching using a blended approach (i.e. a mix of FTF and online learning) and had experienced benefits of the blended mode. There was interest in the impact on his students with the move to ERT, and the possible implications of full online learning for the class in the future. This paper evaluates the impact of ERT on this blended course. A descriptive case study approach, using qualitative methods and applying the Context, Input, Process and Product (CIPP) evaluation model proposed by Stufflebeam (2003) and Stufflebeam and Zhang (2017) was applied. The context analysis involved an assessment of needs, resources and general environment for the course. Input analysis included assessing strategies used in the blended and ERT approaches, while process evaluation examined the extent to which course outcomes were met. Product evaluation examined the intended and unintended outcomes of the ERT course.

Reasons for conducting the study

Prior to 2016, the course lecturer noted with concern issues of poor class attendance in the FTF sessions, along with loss of focus at sessions. This was the only Year 2 course in a 3-year degree programme, scheduled on a Friday and it was felt that the delivery mode of the course and loss of focus was the reason for poor attendance. The lecturer took the initiative to change the delivery mode to see if it would benefit these students. The lecturer started delivering the course in a blended format from 2016, using a combination of F2F teaching and online learning through video recordings and asynchronous discussions. At the onset of Covid-19 it was an easy transition to deliver the course totally online as students were already familiar with working in an online mode. It was felt that perspectives from students and the lecturer on the impact of the transition from blended to ERT would positively inform future design of the course and provide useful information for other higher education lecturers involved with ERT.

In a broader context, based on the University’s Strategic Plan, 2017–2022, it was felt such a study could inform strategic planning for the institution as well as other higher education institutions.

Research questions

The main research question was: What was the impact of the hasty transition from blended learning to ERT on students and teachers? Other questions considered were:

How was the transition from blended learning to ERT facilitated by the institution? (Context)

What were the benefits to students and lecturer of the blended approach in the course? (Process)

How did the blended course design help the transition to ERT? (Input)

What were the benefits to students and lecturer of the ERT approach? (Process)

What were the challenges experienced by students and lecturers in moving from Blended Learning to ERT? (Context and Product)

What were the perspectives of students and lecturer on the requirements for offering the course totally online in the future? (Product)

Literature review: blended learning, online learning, ERT and the CIPP model of evaluation

Blended and online learning

The term “blended learning” has been defined in different ways in the literature and used interchangeably with “mixed mode learning,” “hybrid instruction,” and “technology-mediated/enhanced learning” (Wang et al. 2015). The most common definition of the term “a combination of FTF instruction and computer-mediated instruction” (Graham 2006) was used in this paper. In 2006, Charles Graham described blended learning as the future of classroom teaching and learning, with several benefits to both teacher and student including improved effectiveness, increased access, convenience and greater cost effectiveness. Since that time, blended learning has become a popular approach in higher education and the idea of merging a traditional class with electronic learning is no longer new (Boelens et al. 2017). There have been several studies on the effectiveness of blended learning, and in particular, its advantages over traditional FTF teaching and learning. Evidence indicates that its effectiveness and validity as a new form of learning has been established in practice (Wang et al. 2015).

Over the past 2 decades, different models and frameworks for understanding the elements that affect blended learning from a contextual as well as a pedagogical perspective have been proposed (Wang et al. 2015; Rembach et al. 2019). One such framework is the Octagonal Model proposed by Khan (2003). Badrul Khan’s octagonal framework serves as a guide to plan, develop, deliver, manage, and evaluate blended learning programs from a macro-perspective. It highlights institutional, pedagogical, technological, interface design, evaluation, managerial, resource support and ethical dimensions that can impact effective blended learning. This framework is aligned with an evaluation of blended learning using a context, input, process and product approach since all the dimensions are recognized within these spheres. Another framework of relevance to evaluation of a blended course or programme is provided by Twigg (2003) who classified courses into one of four models: supplemental, replacement, emporium, or buffet. The supplemental model retains the basic structure of a traditional, FTF course but adds a minimal amount of technology based on out of class activities. This may include adding materials, which necessitate no interaction, to a learning management system. The replacement model involves a reduction in class meeting time. Online interactive activities replace FTF time in this model. In the emporium model all class meetings are eliminated and replaced with a Learning Resource Centre featuring online materials and on-demand personal assistance. The buffet model offers students a range of different options that match their individual learning styles.

Twigg (2003) identified blended learning as part of a continuum moving from access to materials online with no interaction (supplemental model) to the replacement model where FTF classes are replaced by online interaction of some sort and then fully online courses with little or no FTF interaction. Fully online refers to a model where there is full online course delivery with the assistance of course support personnel. Bates (2019) takes this idea further by identifying several designs of blended learning including:Bates’s definition of the Flipped model of blended learning is a lean version of the many definitions proposed. The flipped approach is when videos or lecture captures are viewed at home and what was considered home-work in the traditional classroom is done in class. le Roux and Nagel (2018) make the point that the flipped approach is not new. These authors identify several benefits of the flipped approach including improved learner autonomy and enhanced student–teacher and student–student interaction. There has been little research on the effectiveness of the flipped model (Abeysekera and Dawson 2015). Mason et al. (2013) compared the effectiveness of the flipped approach with the traditional approach in an engineering course and found that student performance improved and there was greater satisfaction. They felt that an inverted classroom allowed for more time for learner activities and independent study, and as such it played a key role in modern engineering education (p. 434).

Technology-enhanced learning, where technology is used to support teaching and learning in a classroom setting such as polling software in a lecture or online journaling in class.

Teaching and learning supported by a learning management system, which is used to share learning materials and information and to allow for communication and submission of assignments. Much like the supplemental model of Twigg (2003), teaching in this blended design is FTF.

Bate’s flipped model of blended learning involves the use of lecture capture to facilitate teaching and learning. Students are provided with video recordings of lectures and attend FTF sessions for discussion or class activities.

In the Hybrid or flexible learning model the course is redesigned so that students can do the majority of their learning online. This is likened to Twigg’s replacement model where there is less FTF teaching time, with students only completing FTF activities such as labs that cannot be done online.

Fully online learning, according to Bates (2019), involves no classroom or on- campus teaching. In this model online learning is synonymous with distance learning.

It has been observed that about 92% of all distance and online education studies are at least as effective, if not better, than traditional education (Nguyen 2015). About 3% of the studies compiled by the organization http://www.nosignificantdifference.org show that the traditional FTF format is more effective and about 4% show mixed findings. However, given the issues of selection bias that later studies pointed out and the lack of rigorous methodology with earlier studies, it is difficult to say how meaningful the numbers really are (Nguyen 2015).

Emergency remote teaching

In a time when many countries were already moving more and more toward blended and online learning, the world was impacted by the Novel Corona Virus, Covid-19, which precipitated sudden, drastic changes in the delivery mode of teaching. While the temptation for many leaders in higher education institutions was to laud the move to online learning as a blessing in disguise, Hodges et al. (2020) warned against it. They pointed out that what is being done is not online learning but rather Emergency Remote Teaching, which they define as “a temporary shift of instructional delivery to an alternate delivery mode due to crisis circumstances” (Hodges et al. 2020, p. 17).

They make the distinction between Emergency Remote Learning (ERL) and ERT. ERT captures the simple act of sharing information and knowledge and the actions of teaching which comes before learning can take place. They argued that online learning is systematically planned with preparation and development time usually taking between six to nine months before the course is delivered while ERT is sudden and at times ad hoc. They state that ERT should be separated from online learning and institutions should evaluate ERT to assess any gains for future development. This is particularly relevant for any institution that was making efforts to enhance blended and online course delivery. They propose using a more broad-based evaluation approach such as the CIPP to yield more meaningful results than a student evaluation.

The CIPP model

The CIPP model is the most appropriate for this study because it focuses on improvement and accountability (Brewer 2009). The Context, Input, Process and Product (CIPP) evaluation approach proposed by Stufflebeam (2003) is a comprehensive framework for “guiding evaluations of programmes, projects and systems” (Stufflebeam 2003, p. 31). As the name implies, this model allows for four ‘types’ of evaluation in relation to an entity. The Context evaluation involves assessing needs, problems, assets and opportunities. Input evaluations assess such elements as staffing, budgets and those components that best help to meet needs of the beneficiaries of the programme being assessed. Process evaluations look at outcomes or the extent to which the programme was carried out as intended. Product evaluation looks at the impact of the intended and unintended outcomes of the course or programme. Stufflebeam and Zhang (2017) state that CIPP evaluations address four fundamental questions, What needs to be done? How it should be done? Is it being done? And did it succeed? The answers to these questions can inform planning. While the CIPP approach can involve use of an Evaluation Model Checklist, Stufflebeam and Zhang (2017) note that “at its most elementary level the CIPP model is a commonsense approach” (p.21). The model is being used in its most basic application in this study as a systematic approach to assess the impact of ERT on a blended course on the St Augustine Campus. The study was motivated by an engineering lecturer who had been teaching using a blended approach and had experienced benefits of the blended mode. There was interest in the impact on his students with the move to ERT, and the possible implications of full online learning for the class in the future.

Method

Design

A case study design was used. The case or unit of analysis (Yin 1994) is the Manufacturing Technology course in the Engineering Faculty of The University of the West Indies, St Augustine Campus. Qualitative data was collected from (i) open-ended responses to student evaluations from three cohorts of students; (ii) online focus group sessions with one cohort of students who experienced both the blended approach and ERT and (iii) the reflections of the lecturer and teaching assistants (TAs). The online focus group sessions and reflections were selected because the study entailed examining perspectives of participants and social distancing policies did not allow FTF interviews.

The context of the participants, the sample and the method of data collection will be further explored in the following narrative.

Participants

Context

The St Augustine Campus, where the Manufacturing Technology Course originates, is one of four landed campuses of The University of the West Indies. This five-campus higher education institution, with one virtual campus in the Caribbean, caters to 50,000 students in 17 English-Speaking countries. Each campus was encouraged to explore online learning in an effort to increase access thoughout the network. Covid-19, however, forced all four landed campuses to transition to ERT.

The Manufacturing Technology course—from blended to ERT

For the 2016/2017 academic year, the Manufacturing Technology course was delivered as a blended course using a flipped classroom technique for the first time. Thirty-minute average Screencasts of course topics were developed by the lecturer and made available to the students in the Learning Management System (MOODLE), along with relevant course material and a discussion forum. Students were asked to study, using the Screencasts in their own time and at their own pace, prior to the live class. In the live classroom, the content and difficult concepts were reviewed for 30–60 min. The class was then divided into groups which engaged in FTF activities based on the topics in the Screencasts. The groups were subsequently provided with the grading scheme for the answers and feedback from the TAs and Lecturer. This was in an effort to prepare them for the formal assessments.

In the 2018/2019 academic year the course was further blended by replacing a percentage of the FTF sessions with online synchronous sessions using Blackboard Collaborate (BBC) web conferencing software. The flipped approach was continued with the use of screencasts to initiate class activities and discussion. There were 2 FTF and 10 online, synchronous sessions. In the online sessions, TAs assisted the course lecturer with marking the responses of the group in-course activities based on the screencasts. In Semester II of the 2019/2020 academic year, teaching shifted to ERT mode.

Samples

Data were collected from:

Three separate cohorts of students in the second year of the three-year undergraduate mechanical engineering degree who were pursuing the Manufacturing Technology Course and had completed the Student Evaluation of Courses and Lecturers (SECL) survey. The cohorts were drawn from Academic Year (AY) 2016/2017 (29 students), 2018/2019 (27 students) and 2019/2020 (38 students). The students were a mix of male and female between the ages of 18 and 25.

One lecturer and two TAs for the Manufacturing Technology Course

Procedures

Although the course design changed with regard to the nature and percentage of blend of online and FTF interaction, the course content as well as the course lecturer remained the same throughout the 2016/2017, 2018/2019 and 2019/2020 academic years.

Qualitative analysis has been defined as a research method for the subjective interpretation of the context of text data through the systematic classification of coding and identifying themes or patterns (Hsieh and Shannon 2005). Context, input and product assessments were done using the following qualitative data:

Data gathered from the open-ended aspect of the student evaluation for AY 2016/2017, 2018/2019 and for 2019/2020. For AY 2016/2017, the response rate for the open-ended questions was 22/38 students (57.89%); for AY 2018/2019 the response rate was 23/30 (76.67%) and for 2019/2020 it was 81.58% (31/38). The open-ended questions were coded according to data-driven coding, i.e. categories were developed based on the responses to the questions. The three open-ended questions were:

What did you like best about the course?

What did you like least about the course?

How do you think this course could be improved?

Online focus group sessions using the chat feature in Blackboard Collaborate, a web conferencing software used by the university was conducted with students in the 2019/2020 cohort. The course lecturer posted the questions in the synchronous online chat and saved the responses of the students. Responses were received from 20 students (See Online Appendix 1 for Focus Group Questions with Students)

Reflections of the main course lecturer and the TAs conducting the lab sessions were collected. (See Online Appendix 1 for Reflection Questions for Lecturer and TAs).

Using thematic analysis, qualitative data from the student focus group sessions and reflections of lecturer and TAs were categorized according to the themes experiences of online learning, preparedness, the learning environment, communication and support received from the university, and requirements for offering the course in the future

Table 1 indicates the types of evaluation research questions and the data collection method.

Table 1

Types of evaluation undertaken with associated research questions

Type of Evaluation & research question	Research Method and procedures
Context 1. What was the perception of the students of the blended course prior to the pandemic? 2. How was the transition from blended to ERT facilitated by the institution? (context) 3. What were the challenges experienced by students and lecturers in moving from Blended Learning to ERT?	• Data from Student Evaluation of Course and Lecturers, Open-ended questions • Data from focus group sessions with students doing ERT • Data from reflections of lecturer and TAs
Input 1. How did the blended course design help the transition to ERT? (Input)	• Data from focus group session • Data from reflections of lecturer and TAs
Process 1. What were the benefits to students and lecturer of the blended approach in the course? (Process) 2. What were the benefits to students and lecturer of the ERT approach? (Process)	• Data from focus group sessions
Product 1. What were the perspectives of students and lecturer on the requirements for offering the course totally online in the future?	• Data from focus group sessions and lecturer and TA reflections

Data analysis and results

Context analysis

The context analysis involved assessing needs, problems, assets and opportunities. The research questions to be answered were: What was the perception of the students of the blended course prior to the pandemic (2016/2017 and 2018/2019)? How was the transition from blended to ERT facilitated by the institution? and What were the challenges experienced by students and lecturers and ease of moving from Blended Learning to ERT?

What was the perception of the students of the blended course prior to the pandemic (2016/2017 and 2018/2019)? [Drawn from data from Student Evaluation of Course and Lecturers, open-ended questions]

What the students liked best

Students who completed the blended course prior to the pandemic liked the convenience of the pre-recorded lectures via the screencasts and the convenience of having some aspects of the course available online. In AY 2016/2017 the approach was a flipped one, with the course being mostly FTF, with materials, including Screencasts available in the Learning Management System. With this approach, eleven of the 22 respondents mentioned the screencast was liked best. In AY 2018/2019 the course was further blended by replacing a percentage of the FTF sessions with online synchronous sessions using Blackboard Collaborate (BBC) web conferencing software. In the 2018/2019 evaluation, sixteen out of twenty-three responses indicated that the availability of the course online was what they liked best. The words ‘comfort and convenience’ were coded six times. Students also mentioned benefits of not having to find parking spaces, leave their home on a Friday and having access to the recorded content. The student satisfaction seemed to stem from their hassle-free, limited distraction in the learning environment.

In AY 2016/2017, where there were more FTF sessions, the students also liked the interaction involved. The words ‘interaction and interactive’ were coded a total of eight times out of twenty one responses to the question “what did you like best in the course?”. These words were however only mentioned once in twenty-three responses in AY 2018/2019, which had less FTF sessions and more synchronous sessions done via blackboard collaborate. Laurillard (2000) argued that a university education must go far beyond access to information or content and include “engagement with others in the gradual development of their personal understanding”. Interaction with students is closely linked to motivation in the subject area (Maharaj et al. 2018).

What students liked least

Timing and scheduling were the major hindrances identified by the students of AY 2016/2017 when asked what they least liked about the course. Nine of the 17 responses mentioned the inconvenient time slot, i.e. Friday evening. Four responses focused on the workload stating that there was too much content. Other responses included that the audio quality of the screencast was poor and a need for a more practical component. Changing of the day and time of the class was frequently mentioned to improve the course. The implementation of the online format resolved this issue as no student complained about day and timing in the 2018/2019 survey. This therefore leads us to believe that the underlying problem was not the date and time of the class but the inconvenience of attending class in general. In the 2018/2019 evaluation most of the students disliked the group work and mentioned the heavy workload.

How was the transition from blended to ERT facilitated by the institution? [Drawn from data from focus group sessions with students doing ERT and data from reflections of lecturer and TAs]

The lecturer noted there was support for online learning by the institution and some level of preparedness in the infrastructure such as an established learning management system (MOODLE) and facilities for synchronous delivery through Blackboard Collaborate. The Campus’s Centre for Excellence in Teaching and Learning (CETL) provided on-going training in online learning for lecturers although not availed by all lecturers.

There was a perception by students and lecturers that the communication and support received from the University was insufficient. While there was acknowledgement that the institution had some support structures for blended and online learning, the perceptions of the course lecturer and TAs on the transition were mixed. One TA felt that the technology infrastructure was sufficient to handle needs of lecturer and students.

One TA stated that the campus should have provided high speed Internet to lecturers and students. It was pointed out that course lecturers were trained to use the online platforms while TAS were not. The lecturer expressed his views as follows:

Previous support from CETL (both BBC training and Screencast development) was provided. Yes, the BBC was there and available. Associate Professional staff was previously available to help me edit the screencasts I produced. Ideally the campus should have provided high-speed Internet access to both lecturers and students or at least reliable access.

Communication and support received from university

Students’ views on the communication and support received from the university were mixed. Some students stated that communication from the university took too long. Other students felt that teaching staff were appropriately responsive:

GR: “I think the Dean and lecturers and everyone were understanding and they really listened and communicated as much as they could”

The TAs stated that they received their communication directly from the lecturers, with no direct communication from the institution indicating the possibility of gaps in communication. This was supported by the Lecturer who commented on the slow communication flow:

Lecturer: “Personally, I think the communication was effective and clear to the students. Some information was in flux at times such as the recommended mode of final exam format (take-home exam, online quizzes on Moodle etc.).”

With regard to support all respondents stated that they did not receive any new or additional support, and relied on their familiarity with blended learning and navigating the online platform to aid them in online only learning.

The mixed perceptions of the lecturer and TAs and students suggest that while there was some support for the move to online learning, the level of preparedness may not have been consistent. This suggests that provisions for training and use of technology in teaching and learning was not universally accessed or applied throughout the campus. Students’ experiences in courses varied.

The lecturer expressed his view on what could be done to make a transition to online learning easier. He suggested including a baseline survey to determine those who seemed to be struggling the most with adapting to ERL and then to identify the support needed and provide it as far as is possible. This support might include:

…computer software education (Windows and MS Office tool training) with access to this software and associated hardware (such as laptops...)”

What were the challenges experienced by students and lecturers in moving from Blended Learning to ERT? [Drawn from data from focus group sessions with students doing ERT and data from reflections of lecturer and TAs]

Learning environment

The learning environment refers to more than just the “classroom” or specific place where teaching takes place, but also includes characteristics of the learners; the goals for teaching and learning; the activities that will best support learning; the assessment strategies that will best measure and drive learning; the culture that infuses the learning environment (Bates 2019). The students’ learning environments changed with the transition from either FTF or blended learning to the online only format of teaching. Many students who relied on the university facilities such as Internet access, computers, the library or an overall quiet environment no longer had access. Challenges in the learning environment surfaced when students were asked what additional resources they needed to do their course online. Students complained about poor connectivity or lack of connectivity:

MB: “It was easy to transition but circumstances are different where I am now (bad internet)”

Many students mentioned that the change in their learning environment from a physical to remote one was a very difficult adjustment. They complained of having noisy and distracting home environments. Three students agreed that having access to “a quiet room” would have been helpful. In addition, familial obligations and chores affected students’ ability to be productive at home.

SR: “yes and at the same time there are demands at home too like chores that have to be done”

GR: “then having to help younger siblings who are also at home while essential parents at work”

Students stated it was more difficult to concentrate on schoolwork at home. In addition, one student stated that access to the university library physical space would have been a helpful resource. A few other students shared these sentiments.

The teaching staff also highlighted the technical issues experienced by the students. The course lecturer stated:

Lecturer: “There were personal issues with ISP where connectivity would be lost at times and students would have to wait a few minutes for me to come back online. Students’ connectivity would drop at times as well. One student complained that he didn’t have a good Internet connection.”

The learning environment as part of the context changed dramatically for both students and teaching staff and required adjustments which may not have been sufficiently supported by the institution based on the responses received from students. In the previous environment even though students were using blended learning, they still had the option of the physical facilities on the campus that provided quiet places for working and access to computers and the internet when needed. The sudden shift to spaces not conducive to study resulted in discomfort for students and TAs.

Input analysis

Input analysis looked at the components that best help or meet needs of the beneficiaries of the programme being assessed. An attempt was made to assess which elements of the existing blended course helped or benefited the students and lecturers during the transition to ERT.

How did the blended course design help the transition to ERT? (Input) [Data from focus group session with students and from reflections of lecturer and TAs]

Advantage of the Blended Experience

All the students in the course felt the experience of the blended course adequately prepared them to switch from blended learning to online learning. They felt that prior knowledge of the technology and platforms used for online learning gave them knowledge of what to expect and how to navigate with ease. The word most commonly used by the students in the focus group was that their transition was “seamless”.

The course lecturer also described the transition as “seamless”. He had received previous training and had been using the learning management system and web conferencing software for his blended courses prior to the shift.

Process evaluation

Process evaluations looked at the extent to which the programme was carried out as intended and what were some of the benefits.

What were the benefits to students and lecturer of the blended approach in the course? (Process)? [Drawn from data from Student Evaluation of Course and Lecturers, open -ended questions, Focus Group with the students and from reflections of lecturer and TAs]

As noted earlier from qualitative data collected from AY 2016/2017 and 2018/2019, the main benefit of the blended approach for the students was the convenience of not having to physically come to class.

The lecturer noted that more students achieved at least the minimum standards to pass the course when it was delivered totally online. He compared the results across the three academic years and shared in his reflection that the average in-course exam mark was almost identical in 2016/2017 and 2018/2019, being 16. 2 and 16.1, respectively, out of a total of 25 marks. However, the average in-course mark for the 2019/2020 academic year was 18.3 out of 25. The final exam mark 66.9 out of 75 in 2016/2017 compared to 67.4 in 2018/2019 and 72 in 2019/2020. Four students obtained a mark below the passing grade in the course in both academic years 2016/2017 and 2018/2019 whilst no student failed in the 2019/2020 academic year.

What were the benefits to students and lecturer of the ERT approach? [Drawn from data from Student Evaluation of Course and Lecturers, open-ended questions, Focus Group with the students and from reflections of lecturer and TAs]

While students indicated challenges with the online environment, in the evaluation for AY 2019/20 what they liked most about the course was the online sessions. These they described as interactive, and better than what pertained in their other courses. In the previous offering of the course, students did not find synchronous online sessions particularly interactive. One possible reason for this was the change from student group tutorial break-out sessions, to full class online tutorials conducted by the lecturer. This was done because the student break-out group tutorials were found to be ineffective.

Fifteen of the 31 students responding to the question “what did you like best about the course” in the 2019/20 evaluation indicated that the online environment was what they liked best. Seven respondents indicated that the interactivity of the course was best while other respondents liked the lecturer’s teaching style and the interesting presentation of content.

The lecturer felt that students benefitted from ERT. He noted that the spread of course marks was less in ERT mode in academic year 2019/2020. This could be attributed to the change of the exam format from closed-book, less open-ended, to open-book, more open-ended. This was to facilitate students sitting the exams from their homes in a non-proctored setting. A non-proctored setting could also provide opportunities for collusion.

Product evaluation

Product evaluation looked at impact—what were the intended and unintended outcomes of the course or programme. One research question was asked in relation to this aspect of the evaluation, namely: What are the perspectives of students and lecturer on the requirements for offering the course totally online in the future?

What are the Perspectives on Offering the Course online in the future? [Drawn from data from Student Evaluation of Course and Lecturers, open-ended questions, Focus Group with the students and from reflections of lecturer and TAs]

With regard to the question “how do you think the course could be improved” in the 2019/20 AY evaluation, seven of 22 respondents indicated the importance of having a practical element via arranged site visits or relevant videos. Four of the respondents felt there was need for scheduled tutorials separate from class time and two respondents mentioned using better equipment and online design.

Respondents in the focus group session indicated that more tutorials separate from the regular classes would enhance the course in the future. With regard to preparing for future online courses in general the students felt they should be provided with sufficient information and resources to support online learning.

Perspectives of TAs and lecturer

Feedback from the TAs and lecturer indicate that data should be collected from the experiences moving to ERT to inform planning.

TA 1: “Perhaps there could be faculty meetings on ERT reflections. Surveys can be given to students online to elicit their thoughts and reflections on what occurred. It could be collated on a campus level and delivered to staff and students with discussions on the implications for the future”

A key requirement identified was adequate technical support for staff and students and reliable internet access. For this course in particular, the availability of online lab activities was noted.

Conclusions and recommendations

This paper sought to evaluate the impact of ERT on a blended course in engineering applying the Context, Input, Process and Product (CIPP) evaluation model.

The context analysis highlighted the need for more consistent training in the use of technology for teaching and learning, and for teaching staff at various levels. It also pointed to a need for more technical support for teaching online, and greater access to broadband Internet access for staff and students. The need for more timely communication that filters to all levels of the institution and the development of a learning environment for the creation of a community of learners were also identified as critical, including arrangements for tutorial support and practical activities especially in Engineering courses.

Schiavio et al. (2021) affirmed the need for a community of learners through their observation that students, during the ERT period sought connections beyond the teacher possibly due to the lack of FTF interaction. These needs should be acknowledged and addressed with long-term plans if the institution intends to continue moving towards online learning. Contingency plans should also be made, bearing these needs in mind for any other unplanned event while COVID-19 is still very much a part of the landscape.

Students appreciated the availability of recorded lectures and labs that allowed them to study at their own pace. This is confirmed by Noetel et al. (2021) who found that adding edited recorded video to existing teaching led to strong learning benefits. This was attributed to the learner having more control over their learning. The feedback on the blended courses revealed that students liked the option of working remotely and felt there was interaction. In response to the transition there seemed to be a contradiction with students expressing difficulties focussing in the online environment. This could have been due to the change in their home environments with less privacy since everyone was forced to self-isolate at the same time. The feedback from the students was generally positive regarding the design and delivery of the course. The contradictions which emerged suggested a need to review the current design of the course. The new design could incorporate the Community of Enquiry model (Garrison et al. 2000) which outlines benefits to educational experience of the convergence of social presence, teaching presence and cognitive presence. There may be the need for a design which encourages more active engagement online either asynchronously, synchronously or both. This will require more specific feedback from students regarding course design elements. There is also a need for inputs by the institution such as adequate communication on requirements for online learning and student support for devices, possible safe spaces for learning, tutorial and help/desk support.

Learning outcomes of the course seemed to be met based on the lecturer’s responses that all students passed the course and more did better. He was unsure if the results were due to the change in the examination structure. This suggests that the approach can lead to achievement of outcomes. The issue of promoting excellence or deep learning require further research as well as appropriate assessment strategies.

Product evaluation of the course offered as a blended course and as ERT suggest that there is a preference for the blended or hybrid approach mainly because of the preference for a learning environment that allows for social interaction and use of facilities on the campus. This begs the question of whether the preference is largely influenced by orthodox notions of what a university experience should be. The demographic for this research included traditional university students between the ages of 18 and 25. The preference for a blended option might be influenced by factors other than the design of the course and some of the negative experiences expressed with ERT. There is the need to examine the current design of the course and to approach re-design from the perspective of online learning rather than ERT and planning for online learning up front. Arasaratnam-Smith and Northcote (2017) affirm this, noting a distinction in strategies for FTF classrooms compared to the online learning environment especially with respect to fostering community.

Study limitations and closing comments

This study was not longitudinal and thus does not discuss/address the changes in maturity that occurs in a student as he/she progresses in a program. This maturity may be associated with increased ERT satisfaction as documented by Chen et al. (2021). The motivation of a self-directed online learner (curiosity, interest, and intrinsic based self-improvement) (Bonk et al. 2015) is noted though individual motivational drivers were not required to answer the research questions. Higher attrition rates associated with online learning as discussed by Stone (2019) and García-Alberti et al. (2021) could not realistically be observed in this study as it was not a longitutidanal study. Nevertheless, it was noted that high attrition rates can possibly be mitigated by offering services (personal counselling and mental health/career guidance) to the online students. Lee et al. (2021) commented on the outstanding resilience of the students during ERT, especially those who created customized learning experiences. This augurs well for future design of the course.

Many of the students in the course evaluated described the shift to a fully online mode of delivery during the pandemic as “seamless”. Their participation in blended courses prior to the pandemic contributed to this. This study highlighted that a move to ERT can facilitate teaching and learning during unplanned events, albeit with challenges. However, to maximise the teaching and learning experience for students and lecturers it is important to cater for the components necessary for online delivery. This requires analysis, planning, resourcefulness and agility by the institution and individual lecturers. There must be relevant course design or re-design, ensuring interaction, timely and appropriate communication, and the development of a community of learners.

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 120 kb)