Using Evaluative Data to Assess Virtual Learning Experiences for Students During COVID-19.

Challenge Statement

Our diverse, Hispanic-serving institution (HSI) consists of approximately 64% underrepresented minorities, 45% first generation college students, 14% military affiliated students, and 39% transfer students. During a typical semester, nontraditional students may face demanding challenges outside of their academic responsibilities including full time employment, long commutes, and family obligations. During an emergency or disruptive situation, such as the COVID-19 pandemic, the effects of these challenges may be magnified. Faculty members transferring curriculum online in a short period of time need to understand the best mechanisms for supporting student learning and well-being.

During the 2020 Spring Break week, the university made the decision to migrate courses online and encourage students to move off campus, if possible. Students, as well as university faculty, were informed that an extra week without instruction would be allotted for faculty to prepare their courses for online instruction. While faculty were provided support and training for online course delivery, individual faculty members were allowed significant flexibility in determining their online course structure. For example, asynchronous course delivery was encouraged but not required. For the transition, faculty also had access to a variety of conferencing software for course instruction, but one uniform platform was not required. Selecting Pass/Fail grading was available for a course only if both the faculty member and the student selected this option.

These flexible options resulted in a broad range of course structures over approximately two months of online content delivery. While our Department organized multiple sessions for online learning strategies and best practices, faculty were able to choose their preferred structures. As a result of these varied options, the Department sought to evaluate student perceptions of online course delivery, identify challenges faced during the transition to online learning, and determine student preferences of best practices for remote learning at an HSI with a large, majority-minority, first-generation student population.

Novel Initiative

A survey was developed to collect information from students in the Department, which houses two engineering programs—Biomedical Engineering and Chemical Engineering. The process was reviewed and granted an exemption from the university Institutional Review Board, and the anonymous survey was implemented after completion of the semester. The primary goal of the survey was to identify student perceptions related to the effectiveness of educational activities during the online course delivery. One hundred and seventy students completed surveys (49% Departmental response rate). The students were asked to evaluate only the courses they enrolled in resulting in 352 total course evaluations (an average of 2.1 courses evaluated per student). Twenty-one (21) Biomedical Engineering courses and ten (10) Chemical Engineering courses were included in the survey. Students’ responses to the following four open-ended questions were analyzed inductively, and emergent themes were defined and reviewed.1,2 The themes and number of responses per theme are shown in Tables 1, 2, 3, and 4. The data in Tables 1, 2, 3, and 4 present the most common themes. Themes with the number of student responses ≥ 5 are presented in the Tables.

Table 1

Themes and student response frequencies to survey question 1.

What was the biggest challenge you faced during the online course delivery?, n=352
Theme	n
Self-guided learning (motivation/focus, time management, personal habits)	70
Coursework (grading, groupwork, expectations, getting help/communicating, adapting to online delivery)	69
Technical issues (limited resources, equipment, software, internet)	21
Moving home and family challenges	20
Physical support/face-to-face interaction	13

Table 2

Themes and student response frequencies to survey question 2.

What aspect of the online course content did you feel was the most effective?, n = 352
Theme	n
Faculty availability (office hours, responsiveness)	55
Recorded lectures (record prior to synchronous section, ability to re-watch lectures)	50
Quality or style of lectures	27
Synchronous meeting times for more detail/questions/meeting times dedicated for small groups	26
The instructor’s enthusiasm (commitment & knowledge)	18
Course clarity (due dates, weekly reminders)	13
Accommodating/flexible/acknowledging the situation	11
Online content readily available and timely	10
Including both synchronous and asynchronous options in the course	8
COVID-related supportive activities	7
Examples & problem solving	5

Table 3

Themes and student response frequencies to survey question 3.

What aspect of the online course content did you feel was the least effective?, n = 352
Theme	n
Labs in general (videos not sufficient, need for hands on experiments)	26
Clarity of expectations/timing/process (changing processes during online learning)	20
Problem solving and example problems (online content)	19
Lecture length (lectures longer than original class time, difficult to watch videos in same amount of time)	16
Group projects (difficulty coordinating schedules, communication challenges)	15
Technical issues (video download time)	14
Online in general (general dislike for online content and teaching)	13
Need for more direct faculty interactions	12
Asynchronous (need synchronous lectures)	11
Online content and student engagement	8
Sufficient time for exams	8
Time limits on content	6
Overall workload	5

Table 4

Themes and student response frequencies to survey question 4.

How could our department and faculty improve online course delivery if required in the future?, n = 352
Theme	n
General
Provide flexible options (pass/fail, students choose how they learn)	9
Organize scheduling (i.e. tests, due dates, office hours, recitation sessions) within the department	7
Grading (post grades quickly, drop lowest test score, provide grades on one platform)	5
Redesign course (consistency across courses, pedagogy for transitioning to online learning)	5
Assignments and learning material
Provide clear assignment deadlines and guidelines including posting assignments and syllabi early, and provide deadline reminders	7
Redesign group projects i.e. individual work with group collaborations, senior design, and weekly check-ins	7
More manageable work and tests, including fewer assignments and quizzes	6
More course resources available for online learning (encourage and incentivize resource use and provide readily available support)	6
Lectures
Better video lectures (institute a minimum audio quality, guidelines and structure for videos; include content on how to solve problems; hold video lectures at the same time as class time; more lectures)	16
Live lectures (longer lectures, ability to ask questions during lecture, use Canvas)	7
Asynchronous lectures and videos (keep videos posted online for viewing)	6
Improve slides (add audio and provide slides through alternate platforms)	5
Faculty interactions
Empathize (accommodate reasonable requests; talk about challenges; express intentional support, compassion, respect and positive engagement)	14
More effective communication (transparency, explicitly communicate expectations, reasonable email response time)	11
Virtual office hours—for concerns and one-on-one questions (use Zoom; provide flexible support; answer questions for all students to see)	10

What was the biggest challenge you faced during the online course delivery?

What aspect of the online course content did you feel was the most effective?

What aspect of the online course content did you feel was the least effective?

How could our department and faculty improve online course delivery if required in the future?

Reflection

In general, students responding to the qualitative questions provided constructive feedback on the online teaching environment and the challenges encountered. Specific themes that were consistent across the survey responses are discussed below.

Multiple Modes of Delivery

In an attempt to recognize the challenges that students faced when moving back home, the university encouraged asynchronous delivery of course lectures. Students expressed an appreciation for recorded content and the flexibility it provided, particularly the ability to re-watch videos as needed as a means to improve comprehension. However, a significant number of students discussed the additional need for synchronous content. The synchronous content brought some feeling of normality to the classes and allowed for questions and faculty interactions—contributing to the critical social component of learning.3,4 One student expressed the importance of the time relating to both their own questions but also learning from other students:

Sometimes I would be fuzzy about something but either did not have a direct question to ask or it was difficult to formulate a question and be understood. During those times, it would have been helpful to hear other student’s questions and comments.

And another stated:

Live lectures keep us more motivated to focus.

Students expressed a particular appreciation for courses that had pre-recorded lectures followed by an interactive component that focused on student questions, problem-solving, or more in-depth discussion. Interestingly, this style identified by the students is generally consistent with a “flipped classroom” approach, which takes the lectures to be watched prior to class meeting, and the classroom space is used for solving problems and asking questions, providing more student-instructor interaction.5 This learning environment helps students develop self-regulation as it pertains to their learning that helps them develop metacognition skills transferable to diverse problem-solving settings.6,7 Additionally, self-guided learning and the ability to return to recorded lectures as needed helps students reach understanding at their own pace, which gives high benefit to struggling students, rather than being dependent on the limited classroom time to quickly learn the content.8

Faculty Availability, Flexibility, and Empathy

Similar to having options for viewing lectures, students appreciated faculty flexibility and accommodations for completing assignments and exams. In general, students expressed concern with courses where the content available online had time restrictions or where assignment deadlines were strictly fixed. More broadly, students appreciated instructors who were accommodating as well as those that explicitly acknowledged the situation and expressed empathy.

Although not explicitly stated, students appeared to identify empathetic faculty members based on their flexibility, availability, and public display of acknowledgement of the challenging situation. Students praised faculty members who worked with them to address the complexities of the transition to online classes. They appreciated compassionate interactions with faculty when they felt their instructors authentically cared about their circumstances.9,10 Faculty can exhibit empathy by “listening”, thinking ahead of ways to provide support for their students, and acknowledging uncomfortable feelings and challenging situations.11 Now, more than ever, the need to train ethical and responsive engineers becomes an imperative goal which can be modeled through an ethic of care.12

Students requested more frequent interactions with faculty in general, including “low-stakes” assignments, virtual office hours, and reminders about deadlines. It appears that students saw the value of in-person classes as multifaceted. While online office hours were appreciated, students more often identified the positive impact of rapid response via email or readily available communications via online discussion groups (e.g. Slack).

Socio-Emotional Support for Transitioning to Online Learning

Student comments also pointed to the overall challenges and burden of transitioning to distance learning, moving off campus, and the general disruption of the pandemic. Technically, students were challenged to learn to use multiple new software platforms. They also reported having difficulty communicating and receiving quick responses for help with these systems. A feeling of being “overwhelmed” after the transition to online learning was a common response. This may have been attributed to the stress of the pandemic and possibly a result of a type of grief over the unexpected loss of face-to-face academic interactions.

Many of the students were impacted by returning home. This move sometimes resulted in technical issues, including poor internet connectivity. Many lacked proper privacy in their family’s home to study or participate in virtual lectures, at times requiring the shared use of a computer with multiple family members. In addition, being home meant some students needed to assist in the care for younger siblings or relatives as well as helping with household duties. Some also felt the financial strain of moving away from campus and losing critical employment opportunities.

Students also experienced a decrease in their motivation and focus to complete their classwork. When asked, “What was the biggest challenge you faced during the online course delivery?”, one student commented:

Motivation. I’m the type of person that has to go to the library or any other place but my home to focus on studying.

Students’ who identified perceived challenges linked to self-motivation were extracted and could be clustered into four main areas highlighted in Fig. 1.

Figure 1

Perceived challenges with remote learning related to self-motivation.

More than a third of students who responded that self-motivation was the biggest challenge faced perceived the home environment as not conducive to learning, including distracting and not “academic-friendly”, lacking privacy, or demanding their attention. Another 28% of students commented that the online learning environment was related to their challenge with self-motivation, citing the lack of focus to view videos online, attend lectures, or spend the majority of their time on the computer. Twenty percent of students pointed to their own personal struggles with lack of concentration, optimism, commitment, and balance, including time-management skills during the sudden transition. A final theme identified as general academic captures students’ perceived challenges completing academic activities such as group projects, assignments, and attending lectures.

In general, their responses indicated the need for greater socio-emotional support during an unexpected transition along with attention and acknowledgement about the challenges. The small actions proved empathetic and necessary for students to cope with such a marked change.

Laboratory Classes

Perhaps one of the largest impacts on the student learning experience was the lack of access to advanced facilities for laboratories and senior design projects. Facilities include 3D printing equipment, materials characterization tools, advanced microscopy, and a pilot plant scale distillation column. Hands-on experiences with this equipment is important for training engineers and is not possible with at home lab experiments. Options for students included videos of experiments, research papers and study design exercises, and experimental data provided for analysis and interpretation. However, students identified the limitations of these approaches and lamented the loss of the hands-on experience. Students felt that this type of learning was best done in the physical space with the equipment designed to provide this experience.

Group Projects

A final common theme centered around the challenges related to group projects. Students found difficulties coordinating schedules, meeting online with group members, and collaborating with all members in the group. Interestingly, this seems like one of the more important skills that could be developed during online course delivery. Engineering work is increasingly global in scale. While this requires cultural understanding, it also requires an ability to collaborate effectively remotely. Students struggled to successfully collaborate in team settings through virtual platforms. Future developments could focus on empowering students to learn best practices for remote collaboration, including setting clear and shared expectations, collaboratively creating artifacts on a shared platform, and collectively and individually reflecting on their contributions.13 This may require early guidance from an instructor or teaching assistant, but direction to promote successful group work through online methods was needed.

Implementations to Improve Online Teaching and Learning

The transition to online teaching and learning was not expected and perhaps due to this, created unique challenges at a minority serving institution. Students and faculty managed multiple stressors during the transition to distance learning despite limited time for training and transitioning. While student input is not the only information to consider, much of what students identified is consistent with education research and learning theory. This information can provide insights into practices that may be used to enhance the online learning experiences in the future for engineering students. Some options to consider for virtual courses include, but are not limited to, the following:

Multiple modes of learning Provide students with both asynchronous and synchronous modes of learning. Recorded live lectures should be housed online for students to readily access as needed after the session ends. Providing students with additional synchronous opportunities for participation helps students connect more easily as needed. These opportunities can come during synchronous lectures, online forums, small group remote meetings, or interactions with faculty during virtual office hours or online discussion groups.

Flexibility and empathy Faculty should listen, observe, and acknowledge the challenges that students may be facing in challenging circumstances. Students build trust when faculty show care and “feel with [them]” through respectful and empathetic interactions as well as when they are “prepared to care” for students.14,15 Providing flexible opportunities for contributing to the course aid in students’ access to knowledge and show attention, thought, and care for the diverse experiences that students may face in an online learning environment. These options can be tailored to student’s specific needs but should be communicated at the start of the semester/course, if possible.

Communication Set clear objectives, deadlines, and expectations for engagement from the beginning of the course. Whenever possible, use technology to automate course news and upcoming deadline reminders. Course management platforms also have mechanisms for sending personal emails to students identified for a specific characteristic i.e. missing an assignment or falling below a pre-determined course average. Students appreciate communication and transparency along with readily available responses to questions, but this may not always be fair or practical for faculty. Therefore, instructors should manage expectations by communicating a timeline and mechanisms by which students will receive feedback.

Group Projects Group projects are critical to many aspects of engineering education. Faculty can be proactive in training students to effectively participate on virtual teams by leading student groups, for example, to use shared platforms where artifacts can be collected, viewed, and edited synchronously and asynchronously by all group members. Additionally, faculty can help groups identify project goals and milestones spread throughout the semester helping students reflect on their progress and gauge the project status at any given point in time. Therefore, the course should include instruction and guidance on developing working groups through online project management.13

Student feedback is an important tool in evaluating our approach but not the only input of importance. Input should be received from all stakeholders (faculty, teaching assistants, administration, etc.), and the impact on all parties should be considered. It is critical that, as we reflect on student input, we recognize the potential risk for our faculty. Converting classes to online delivery, ensuring both recorded lectures and multiple learning resources are produced, communicating rapidly, reminding students of deadlines, and hosting small group meetings would be a challenge particularly if these elements need to be completed for all courses in a short period of time. One student addressed the incredible work of a faculty member,

I don’t know where he found the time to do all of these but he did and I am thankful for it.

It can take an incredible effort for faculty who, while committed to supporting students, also have other obligations. We must recognize these challenges, provide support for faculty to deliver the content, and assist with maintaining their well-being. Only then will we have the capacity to support our diverse students in their learning.