Impact of COVID-19 on public health nursing student learning outcomes.

BACKGROUND: The COVID-19 pandemic has highlighted the importance of a strong, effective public health nursing workforce while also requiring public health nursing faculty to adapt teaching strategies as courses transitioned online. It is essential to understand how the pandemic-enforced transition from face-to-face to remote learning impacts student outcomes. The purpose of this paper is to compare student learning outcomes in a pre-licensure public health nursing course before, during, and after the transition to remote learning.
METHODS: Descriptive statistics were computed for assignments, exams, and final course grades for three terms (Fall 2019, Spring 2020 and Fall 2020).
RESULTS: Analysis showed statistically significant differences between terms for assignments and exams but not the final course grade. However, these differences were driven by small standard deviations rather than differences between mean scores demonstrating that there was actual little difference in student learning outcomes across terms.
CONCLUSIONS: Authors suggest strategies to support consistent academic outcomes and future research needed understand student learning outcomes during the pandemic; ultimately building the public health nursing workforce necessary to address the current and future public health crises.

INTRODUCTION

As the COVID‐19 pandemic emerged in the United States in early 2020, nursing programs across the country were forced to rethink nursing education. This public health crisis simultaneously elevated the need for a strong public health nursing workforce while challenging faculty who traditionally use face‐to‐face (F2F) teaching to develop this workforce. During the COVID‐19 pandemic public health nurses serve on the frontlines in a variety of roles, highlighting their importance. Public health nurses are leading mobile strike teams to investigate and contain the spread of COVID‐19, interpreting guidance from the Centers for Disease Control and Prevention to best serve underserved populations, and delivering patient‐specific education on infection control measures such as isolation and quarantine (Edmonds et al, 2020). At the same time, public health nursing faculty swiftly implemented teaching strategies to maintain student, faculty, staff and patient safety while continuing to prepare future nurses to enter the workforce (Bejster et al., 2021; Dewart et al., 2020; Morin, 2020; Tomietto et al., 2020).

The American Association of Colleges of Nursing (AACN) released a set of COVID‐19 preparedness and response considerations for nursing programs based on up‐to‐date public health guidance. These recommendations included preparing for the transition of face‐to‐face (F2F) non‐clinical courses to an online delivery format (AACN, 2020). Based on these recommendations, nursing educators had to rapidly transition F2F didactic courses to remote learning experiences. Now, over a year later, as the pandemic continues, many traditionally F2F didactic nursing courses have remained remote. While the initial transition to remote learning was emergent and took place quickly, a more methodical, planned approach to sustain the delivery of online learning is necessary to ensure high‐quality nursing education moving forward (Morin, 2020).

Further, it is important to consider how such a widespread, sustained change in course delivery format has impacted nursing student learning and workforce development (Morin, 2020). While exploration of the transition to remote learning as a result of COVID‐19 is beginning to appear in the literature (Dewart et al., 2020), there remains a gap in examining the impact of this transition on student learning outcomes. The purpose of this paper is to describe student learning outcomes in a pre‐licensure public health nursing course before, during, and after the transition to remote learning in response to the COVID‐19 pandemic.

METHODS

Setting

This public health nursing course is part of the curriculum for a generalist entry master's (GEM), pre‐licensure nursing program, which is designed for students with a bachelor's degree in a field other than nursing. The GEM program is housed in the College of Nursing at an urban, university medical center in the Midwestern United States. The 15‐week course is in the fourth term of the six‐term program.

The public health nursing course uses an ecological approach to examine the health of individuals, families, and populations within the context of the community. Major concepts covered include the community assessment process, designing, implementing and evaluating health promotion activities, social determinants of health, environmental health, upstream thinking, epidemiology, emergency preparedness and disaster management, public health nursing ethics, family health, and public health policy. Students engage with the course concepts in several ways, including assigned readings, a “PHN in the media” course segment that includes a current events discussion, asynchronous and synchronous lectures, and active learning activities. Examples of active learning activities include engagement in unfolding case studies, partner work to role‐play the development of ecomaps and genograms with patients, and small group work to create disease transmission diagrams. Student learning outcomes are evaluated through several graded assignments and three examinations. All graded assignments and examinations are in alignment with course objectives. All assignments and examinations, other than the final cumulative exam, can be considered formative evaluation methods as they allow students to identify areas for continued learning. Learning evaluation criteria are described in Table 1.

TABLE 1

Learning evaluation criteria

Evaluation criteria	Assignment description	Week due	Assignment scoring scale	Percentage of course score
CDC immunization module	Online learning modules with post‐test, asynchronous, individual assignment	11 a , 1	5	5%
Discussion board activity	Online videos, discussion board post with individual reflection and response to one peer, asynchronous, individual assignment	4 a , 11	5	5%
Poster presentation	Development and oral presentation of a scientific poster on a current public health issue, synchronous, b group assignment	12 a , 9	100 a	15%
Synthesis presentation	Presentation synthesizing course concepts and application to the clinical setting, synchronous, group assignment	14	100	12%
Synthesis presentation peer review	Peer evaluation of all synthesis presentation group members, asynchronous, individual assignment	14	5	3%
Exam 1	60‐min exam, synchronous	6	100	20%
Exam 2	60‐min exam, synchronous	10	100	20%
Exam 3	60‐min cumulative exam, synchronous	15	100	20%

Notes when assignments were due during Fall 2019, if different than other terms. .

In Spring 2020, the oral presentation was omitted and the assignment was scored out of 90 points.

The public health nursing course is typically delivered F2F once per week in an active, learning classroom setting. The CDC immunization module and discussion board activity were always completed asynchronously online outside of F2F class. However, other activities and all three exams were conducted in a F2F environment. The course was last taught 100% F2F in Fall 2019. In Spring 2020, this course started just 1 day before a novel coronavirus was identified as the cause of respiratory illness and death in China (Holshue et al., 2020). In the United States, the first case of this new virus was identified on January 20, 2020 (Holshue et al., 2020). Starting on January 27, 2020 (week four of the term), faculty began incorporating emerging knowledge about the novel coronavirus into course content. This was done mainly during a 10‐min section of class titled PHN in the Media, during which faculty would highlight a news story related to the new virus. Further, course faculty used the emerging public health crisis as a learning opportunity, relating the current events to course content whenever possible. The emerging pandemic was discussed in class in relation to public health concepts such as the social determinants of health, communicable disease and ethics.

By early March 2020, the virus was given the name COVID‐19 and public health officials began to issue guidance in response to the growing number of cases in the United States (Centers for Disease Control and Prevention [CDC], 2020). Local public health officials issued guidance limiting the number of people allowed to gather in one place (City of Chicago, 2020). On March 7, 2020, all non‐essential large group gatherings over 50 people at the university were cancelled. However, based on the University guidance, classes continued as they were considered essential. On March 9, 2020, when there were just 647 total cases of COVID‐19 in the United States (CDC, 2020), we held our last live class of the year (week nine of the term). On March 11, 2020 the university cancelled all live classes and asked faculty to make immediate arrangements to continue classes remotely for the foreseeable future. This meant that while the course began F2F during the Spring 2020 term, due to the emerging COVID‐19 pandemic the course transitioned to 100% remote during week 10 of the 15‐week course. The course remained 100% remote for the Fall 2020 term. Faculty were consistent across all three terms. Table 2 shows the transition in course delivery format through‐out the three terms. Details about this transition and student perceptions of the transition can be found in 2021.

TABLE 2

Course delivery format by term

	Fall 2019		Spring 2020		Fall 2020
	F2F	Remote	F2F	Remote	F2F	Remote
Lecture	X		X weeks 1–9	X weeks 10–15		X
Office hours	X		X weeks 1–9	X weeks 10–15		X
CDC immunization module		X		X		X
Discussion board		X		X		X
Poster presentation	X			X		X
Synthesis presentation	X			X		X
Synthesis peer review		X		X		X
Exam 1	X		X			X
Exam 2	X			X		X
Exam 3	X			X		X

Statistical methods

Data was collected from the online learning management system for all learning evaluation criterion. All raw scores for each learning evaluation criterion across three terms: Fall 2019, Spring 2020, and Fall 2020, were converted into percentages to permit comparison. Standard descriptive statistics (mean, standard deviation, and range) were computed for assignments, exams, and final course grades for each term and compared between the three terms. Means were compared using a one‐way ANOVA and graphical analysis to identify differences between terms. This project was reviewed by the authors’ Institutional Review Board and approved as a quality improvement project. All data are reported in aggregate to protect student privacy.

RESULTS

The one‐way ANOVA analysis shows that statistically significant differences exist between terms for all assignments (except the CDC module for which it could not be computed as all scores were identical across the three terms) and exams but not the final course grade. Table 3 includes data for all assignments, exams, and final grades across the three terms. For the assignments, these differences appear to be driven by small standard deviations rather than large differences between mean score. This is evident by the largest difference of any pair being only 2.04 percentage points between the poster presentation in Fall 2019 and Fall 2020, as can be seen on Figure 1. Differences on exam scores between terms are caused by greater variability in Fall 2020, as can be seen in Figure 2. Fall 2019 and Spring 2020 exams were never more than 1.41 percentage points different.

TABLE 3

Student learning outcomes

	Fall 2019 (n = 69)		Spring 2020 (n = 73)		Fall 2020 (n = 69)		ANOVA
	M (SD)	[min–max]	M (SD)	[min–max]	M (SD)	[min–max]	F	df	p‐value
CDC immunization module	100 (0)	100–100	100 (0)	100–100	100 (0)	100–100	—	—	—
Discussion board post	99.2 (0.14)	80–100	99.4 (0.23)	60–100	99.8 (0)	90–100	258.25	2	<.01
Poster presentation	98.3 (1.48)	95–100	97.2 (2.02)	93–100	96.26 (2.44)	91–100	6.58	2	<.01
Synthesis presentation	96.28 (3.26)	90–100	96.48 (1.91)	91.25–100	98.26 (1.79)	96.25–100	14.36	2	<.01
Synthesis peer review	99.4 (0.08)	90–100	99.6 (0.1)	82–100	99.2 (0.19)	70–100	166.41	2	<.01
Exam 1	91.99 (5.57)	75–100	90.32 (6.03)	70–98	95.96 (4.83)	80–100	19.61	2	<.01
Exam 2	91.8 (4.83)	77–100	91.9 (5.35)	77–100	87.6 (4.96)	69.99–94	16.48	2	<.01
Exam 3	94.51 (4.42)	72–100	93.1 (4.69)	77–100	91.86 (6.06)	68–100	4.67	2	.01
Final course grade	94.91 (2.47)	84.85–98.68	94.09 (2.27)	88.92–98.35	94.96 (2.58)	87.74–99.15	2.92	2	.06

FIGURE 1

Assignment scores and final course grade by term

FIGURE 2

Exam scores by term

Assignments

There was no difference between the scores on the CDC Immunization Module across the three terms, with all students earning 100 on the assignment. Scores on the discussion board posts were similar with means ranging from just 99.2 to 99.8 [F(2) = 258.25, p < .01], progressively increasing over time. Low scores on this assignment ranged from 60 to 90, while the maximum score for each term was 100.

Mean scores for the poster presentation ranged from 96.26 to 98.3 [F(2) = 6.58, p < .01], progressively decreasing over time. The low score for the poster presentation ranged from 91 to 95, with the maximum score remaining 100 for all three terms. Mean scores for the synthesis presentation ranged from 96.28 to 98.26 [F(2) = 14.36, p < .01], increasing slightly each term. The low score for synthesis presentations ranged from 90 to 96.25, also progressively increasing over time, while the maximum score remained 100 for all three terms. The mean scores for the synthesis presentation peer review ranged from 99.6 to 99.2 [F(2) = 166.42, p < .01], with the highest mean occurring in Spring 2020. The low score for the peer review ranged from 70 to 90, with the high score for each term remaining at 100. Figure 1 includes graphical analysis of assignment scores by term.

Exams

Mean scores for Exam #1 ranged from 91.99 to 95.96 [F(2) = 19.61, p < .01], with the highest mean occurring in Fall 2020. The low scores for Exam #1 ranged from 70 to 80, with the high scores ranging from 98 to 100. Means scores for Exam #2 ranged from 87.6 to 91.9 [F(2) = 16.48, p < .01], with the highest mean occurring during Spring 2020. The low scores for Exam #2 ranged from 69.99 to 77, with the high scores ranging from 94 to 100. Mean scores for Exam #3 ranged from 91.86 to 94.51 [F(2) = 4.67, p = .01], progressively decreasing over time. The low scores for Exam #3 ranged from 68 to 77, with the high scores for all three terms remaining at 100. Figure 2 includes graphical analysis of exam scores by term.

Final course scores

There was very little difference in final course scores across the three terms. Mean final course scores ranged from 94.09 to 94.96 [F(2) = 2.92, p = .06], with the highest mean occurring in Fall 2020. The low scores for the final grades ranged from 84.85 to 88.92. The high score for the final grades ranged from 98.35 to 99.15.

DISCUSSION

The purpose of this paper was to compare student learning outcomes in a pre‐licensure public health nursing course over three academic terms, before and during the COVID‐19 pandemic. It is important to note that in Fall 2019, the term preceding the COVID‐19 pandemic, the course was taught 100% F2F. In the middle of Spring 2020, the course transitioned from F2F to remote learning, and in Fall 2020 the course was taught 100% remote. Nonetheless, based on our evaluation, there was little difference in student learning outcomes across the three terms.

As noted in the results section, all changes in assignment and examination scores were within a few percentage points. Even so, rationale for these changes should be considered. Discussion board grades increased slightly. Asynchronous discussion board activities are an established method for increasing nursing student knowledge in didactic courses (Hudson, 2014). An increase in discussion board scores over time for this course may be a result of students becoming more familiar with this type of online learning activity. As the COVID‐19 pandemic persisted this type of learning activity became used more frequently in courses than it was used during F2F instruction. Further, this activity facilitated interaction with peers which has been noted as important for nursing students during the pandemic (de Tantillo & Christopher, 2020). Moving forward, nursing faculty should consider the use of asynchronous discussion board activities to facilitate student learning and peer engagement.

Synthesis presentation grades also increased. One reason for this may be that students were provided with multiple platforms to coordinate group work during the pandemic. Prior to the pandemic, students were expected to organize F2F group meetings. However, once this course transitioned to the remote environment, faculty set up virtual workspaces for students, allowing them flexibility in how they met as a group to prepare for the presentation. Flexibility has been identified as a facilitator of learning during the pandemic (Bejster et al, 2021; de Tantillo & Christopher, 2020; Goni‐Fuste et al, 2021). Further, de Tantillo & Christopher (2020) noted facilitation of group work during the pandemic to be especially important as it can increase camaraderie during periods of social isolation. The increase in synthesis presentation scores may be a result of students’ desire and appreciation for working in the group setting during this time of isolation.

Conversely, poster presentation scores dropped slightly. This may be a result of the oral presentation of the poster being omitted during the pandemic. Further, while students were familiar with creating a physical poster, transitioning this assignment to be remote required them to build an electronic poster. This required a skill set they had not previously developed. The need for clear expectations and stability during the pandemic are noted in the literature as necessary to facilitate nursing student learning (de Tantillo & Christopher, 2020; Gillis & Krull, 2020). Changes to this assignment and a disconnect between student experiences with previous poster presentations and faculty expectations on this assignment, may have contributed to decreased scores.

Authors also identified a drop in exam 2 scores during Spring 2020. This is not surprising considering this exam was administered in week 10 of the course; the first week this class was taught remote. Students were required to use an online, live, exam proctoring service of which they were not familiar. Exam anxiety is often cited as a stressor and barrier to learning for nursing students (Quinn & Peters, 2017). While there is limited literature available to explore the impact of online exam proctoring on nursing students (Castano et al., 2021), online‐proctoring was noted as anxiety inducing for college students at large during the pandemic (Elsalem et al., 2020). Further, general anxiety for nursing students related to COVID‐19 is well documented (Savitsky et al., 2020). Combined, these factors may be the reason for a decrease in exam 2 scores.

In order to better understand the reasons for changes in student learning outcomes across the terms, authors suggest a qualitative deep‐dive into the student experience exploring rationale, both academic and personal, that may impact learning outcomes. Authors acknowledge that evaluating assignment and exam scores only considers student learning outcomes in a didactic course and may not be transferable to the clinical setting. Authors suggest an evaluation of clinical skills to determine if students were also successful in meeting clinical objectives consistently during the pandemic to more fully evaluate overall student learning.

Limitations

This paper describes the evaluation of student outcomes in one public health nursing didactic course. Results may not be generalizable to other universities, programs or courses. Nonetheless, considering the limited availability of published literature specifically exploring student academic outcomes during COVID‐19, this paper offers unique insight and suggests areas for future exploration.

CONCLUSION

Continued development of the public health nursing workforce is essential to protect the health of the public during and beyond the COVID‐19 pandemic. Results of this project demonstrate that although theCOVID‐19 pandemic presented a variety of challenges for public health nursing students and faculty over the past year, student learning outcomes remained consistent. Moving forward, nursing educators must continue to evaluate student outcomes in a variety of ways to assess how course adaptations may affect student learning outcomes, especially as more courses begin transitioning back to the traditional F2F format. Further, looking beyond academic outcomes and exploring the overall impact of the pandemic on public health nursing students and new graduate nurses will be essential to ensuring the long‐term stability of the public health nursing workforce.

CONFLICT OF INTEREST

Authors do not have any conflicts of interest to disclose.