Virtual Trainings Effectively Prepared the Public Health Workforce to Support Schools During the COVID-19 Pandemic in California in 2021.

CONTEXT: School closures in California due to COVID-19 have had a negative impact on the learning advancement and social development of K-12 students. Since March 2020, the achievement gap has grown between high-income and low-income students and between White students and students of color. PROGRAM: In November 2020, a team from the California Department of Public Health, University of California, San Francisco, and University of California, Los Angeles, developed the School Specialist training for local health department and state employee redirected staff to the COVID-19 response to equip them to support schools as they reopen. IMPLEMENTATION: A pilot of the virtual School Specialist training was carried out in December 2020, which informed subsequent biweekly half-day virtual trainings. The training consisted of lectures from experts and skill development activities led by trained facilitators. EVALUATION: The objectives of the evaluation of the training were to understand whether (1) knowledge of key concepts improved from pre- to posttraining; (2) confidence in skills central to the role of a School Specialist improved from pre- to posttraining; and (3) course learners who were activated to work as School Specialists felt the training adequately prepared them for the role. The School Specialist training team sent pre- and posttraining surveys to learners between February 8 and May 18, 2021. Of the 262 learners who responded, a significant improvement was seen in knowledge, with a mean score increase of 15.6%. Significant improvement was also observed for confidence, with a 20.1% score improvement seen posttraining. DISCUSSION: Overall, the School Specialist training was shown to be effective in increasing knowledge and confidence in preparation for School Specialist deployment. Adequate training and partnerships for local health department and school staff are critical to keep K-12 students safe and to reduce the learning achievement gap during the ongoing COVID-19 pandemic.

The achievement gap between high- and low-income students and between White students and students of color has persisted for decades and cost the US economy between $310 billion to $525 billion a year in lost productivity.1 Students from marginalized or underserved groups have lower test scores, higher dropout rates, and lower college enrollment.2 In March 2020, California Governor Gavin Newsom ordered the closure of more than 670 of the state's 1000 school districts due to the arrival of the COVID-19 pandemic, impacting more than 5.7 million K-12 students.3 As a result, most school districts transitioned to remote learning. Previous studies have indicated full-time remote learning does not adequately replace in-person learning and has also increased concerns for the development of students' emotional and social well-being.4,5 Existing achievement gaps have grown even wider during the COVID-19 pandemic as children from marginalized or underserved populations have less access to resources to complete schoolwork at home.6 Studies cite “broadband gaps” between Black and Hispanic students and their White counterparts, where the former are less likely to have the high-speed Internet connectivity necessary to complete schoolwork.7,8 Access to a home computer also divides students, where 25% of teens living in households that earn under $30 000 per year lack access to a home computer, but only 4% of teens living in households that earn more than $75 000 per year lack the same access.9 Reopening schools safely in California has become an issue of equity and also an opportunity to build the public health workforce, with practitioners from the communities hardest hit to better serve their communities' needs.5

The governor's office of the state of California identified opening K-12 schools safely during the COVID-19 pandemic as a priority, given the toll remote learning has had on low-income students of color and working parents.3 K-12 schools and school districts turned to their local health departments (LHDs) for assistance on the interpretation and implementation of state public health guidance, given most school-based staff members do not formally have experience or expertise in public health. While some LHDs trained their contact tracing workforce to support schools, not all LHDs in California had dedicated staff to support schools with their COVID-19 response in 2020 and 2021.

The California Department of Public Health (CDPH), University of California, Los Angeles (UCLA), and University of California, San Francisco (UCSF) formed the California COVID-19 Virtual Training Academy (VTA)10,11 in March 2020, with the goal to train a large, diverse workforce rapidly and effectively with specialized knowledge and skills to respond to the COVID-19 pandemic. The VTA offers virtual courses to local health staff, redirected state employees to support the COVID-19 pandemic response in California, and school administrators on COVID-19 response efforts. In November 2020, CDPH tasked the VTA team with training LHD “School Specialists” to support school reopening and collaborate with schools starting in the 2020-2021 academic year. School Specialists are a new cadre of LHD staff members who support schools and school districts with COVID-19 public health guidance interpretation, case, cluster, and outbreak investigations, and data management. While the term “School Specialist” is specific to California, other LHD staff members in other states across the country are also supporting schools as they have reopened. The VTA developed the School Specialist training course with the goal of targeting LHDs and state redirected staff in California who would likely be activated as School Specialists for their assigned LHD.

The School Specialist training is a 2 half-day virtual training consisting of 7 didactic lectures and 4 facilitated skill development activities where learners put the concepts into practice with guided case studies. The curriculum was informed by focus groups with local health staff who sought to understand what their teams were currently doing to support schools and in what areas additional knowledge and skills were needed. The VTA organized the first School Specialist pilot training in December 2020 and has continued to organize trainings twice a month thereafter. We collected feedback from learners during the pilot, and the curriculum was updated and improved in response to the feedback. After participating in the training, it was expected that the state and LHDs would activate learners to School Specialist roles in LHDs. The learning objectives for the course were to:

Comprehend the basic epidemiological principles of pediatric COVID-19;

Describe the basic structure and roles and responsibilities of schools and LHDs;

Identify and navigate the essential federal and state policies and guidance for COVID-19;

Practice how to respond to basic questions from schools and know when to triage to other members of the LHD, including using decision trees;

Comprehend the basic principles of outbreak prevention and data management using the statewide COVID-19 data system, California Confidential Network for Contact Tracing's (CalCONNECT's) Shared Portal for Outbreak Tracking (SPOT); and

Practice how to communicate effectively with school district staff.

As added context for the course curriculum, decision trees are infographics that show a flow of possible solutions, choices, or paths of action based on certain situations. They were most often used by LHDs and schools in California for COVID-19 cases to depict what to do if a student or staff member tested positive or had been exposed to COVID-19. SPOT is a digital system to report COVID-19 exposures at schools to their LHDs. This digital platform is integrated with CalCONNECT, California's COVID-19 case investigation and contact tracing data management and communications platform. CDPH offered in-depth trainings to LHDs and schools on SPOT, which is why the VTA did not cover the platform in depth as part of the School Specialist training.

The purpose of this evaluation was to assess the impact of the VTA School Specialist training on learners' knowledge and perceived readiness to fulfill roles as School Specialists in LHDs. Specific aims were to understand whether (1) knowledge of key concepts taught in the School Specialist training improved from pre- to posttraining; (2) confidence in skills central to the role of a School Specialist improved from pre- to posttraining; and (3) course alumni who were activated to work as School Specialists felt the training adequately prepared them for the role.

Methods

Population and setting

From February 8 to May 18, 2021, a total of 553 individuals enrolled and 353 attended one of 8 School Specialist online virtual training sessions offered by the VTA. Cohorts prior to February 8 were not included in the study because pretraining surveys were not administered. Participants included staff from LHDs across the state of California and mission-tasked state employees who were asked to pivot to quickly mobilize the public health workforce. Of note, most of the state redirected staff came from departments outside of public health. CDPH conducted outreach to LHDs and state redirected employee distribution lists, with information and registration links for the course.

Data sources and measures

The VTA team sent pre- and posttraining surveys to learners to assess knowledge of core learning objectives and self-perceived confidence in skills pertinent to the School Specialist role. In addition, the team sent a field survey to learners approximately 3 months after course completion to assess activation status and perceived preparedness for activation. The surveys were developed by the VTA training team and the Monitoring and Evaluation Working Group to reflect the unique content of the training and job responsibilities of School Specialists in California. Participant demographics, including race and ethnicity, occupation and employer, educational background, age, gender identity, and languages spoken were derived from course registration data as seen in Supplemental Digital Content (available at http://links.lww.com/JPHMP/A950).

Knowledge assessment

The knowledge assessment (see Supplemental Digital Content Appendix A, available at http://links.lww.com/JPHMP/A951) contained 11 true/false and multiple-choice questions that reflected course learning objectives, including the roles and responsibilities of School Specialists, epidemiology of pediatric COVID-19, federal and state COVID-19 policies and guidance, management of exposure and outbreak data, and strategies for effective communication with school administrators and parents.

Skill self-assessment

Survey respondents were asked to complete a 5-point scale of skill self-assessment (see Supplemental Digital Content Appendix B, available at http://links.lww.com/JPHMP/A952) to rate their self-perceived ability to perform the following 6 skills central to the role of a School Specialist: (a) encourage people to share accurate information about their activities; (b) de-escalate emotion when engaging with an overwhelmed client; (c) interpret data in a spreadsheet of cases (referred to as a “line list”); (d) deduce important questions to ask of a school liaison; (e) understand data sharing regulations to protect student and staff information; and (f) perform School Specialist job duties.

Perceived preparedness for activation

The VTA team sent the field survey via e-mail to all learners who completed the training including a question on whether they were currently working as a School Specialist. Those who were activated were asked to complete the skill self-assessment again and report specific components of the School Specialist training where they felt they needed additional knowledge and/or practice (see Supplemental Digital Content Appendix C, available at http://links.lww.com/JPHMP/A953).

Analyses

Analyses were conducted using Microsoft Excel 36512 and IBM SPSS Statistics for Windows, version 27.0.13 We conducted a pre/posttraining analysis to evaluate training outcomes. Only trainees who completed all questions on both the pre- and posttraining assessments were included in the analyses. If an individual completed the pre- or posttraining surveys more than once, only responses from his or her first submission were analyzed.

For the knowledge assessment, data were analyzed in 2 ways. First, for each individual knowledge question, the McNemar test as was used to evaluate the significance of the change in the proportion of learners answering correctly from pre- to posttraining. Second, a composite knowledge score variable was created as the sum of correct responses to the 11 knowledge questions (ranging from 0 to 11), and the Wilcoxon signed-rank test was used to assess significant differences in the composite knowledge score from pre- to posttraining.

For the skill self-assessment, data were similarly analyzed in 2 ways. First, for each individual skill item, a confidence score (ranging from 1 to 5) was calculated at pre- and posttraining, and the Wilcoxon signed-rank test was used to assess significant differences in confidence from pre- to posttraining. Second, a composite confidence score variable was created by summing the 6 skill self-assessment items (ranging from 6 to 30), and the Wilcoxon signed-rank test was used to assess significant differences in the composite confidence score from pre- to posttraining.

Finally, to understand perceived readiness after activation in a School Specialist role, descriptive statistics were used to summarize learner's confidence in School Specialist skills and whether alumni reported needing additional knowledge or practice in 10 potential School Specialist duties such as school line lists, SPOT, or supporting schools with school reopening plans.

Data and Results

Sample

Among the 353 course participants, 262 (74%) completed both pre- and posttraining surveys and represent the sample for the analyses. Individuals who completed the training requirements were more likely to be state employees than those who enrolled but did not complete the training (Table 1). A total of 60 (17%) individuals completed the field survey. Approximately one-third of the sample identified as White (34%) and just over a quarter identified as Hispanic (27%). The majority identified as female (71%) and had at least a college degree (83%).

TABLE 1

Demographics of School Specialist Learners Who Registered or Completed the Training From February 8 to May 18, 2021a

Demographic Characteristic	Course Enrollees (n = 553), n (%)	Sampleb (n = 259),c n (%)
Employerd
State of California	190 (36)	103 (40)
Local health jurisdiction	306 (57)	147 (57)
Other	37 (7)	9 (3)
Race and ethnicity
White/Caucasian	191 (36)	87 (34)
Hispanic/Latina/Latino/Latinx/Spanish origin	138 (26)	70 (27)
Black or African American	38 (7)	17 (7)
American Indian or Alaska Native	2 (0.2)	1 (0.3)
Asian	70 (13)	34 (13)
Native Hawaiian or Pacific Islander	4 (0.8)	0 (0)
Two or more	45 (8)	25 (8)
Other	2 (0.3)	0 (0)
I prefer not to respond	43 (8)	25 (8)
Age, y
<18	1 (0.2)	0 (0)
18-24	40 (8)	24 (9)
25-34	139 (26)	74 (29)
35-44	109 (21)	48 (19)
45-54	106 (20)	43 (17)
55-64	91 (17)	43 (17)
65+	19 (4)	10 (4)
I prefer not to respond	27 (5)	17 (7)
Gendere
Male	113 (21)	56 (22)
Female	383 (72)	184 (71)
Otherf	7 (1)	0 (0)
I prefer not to respond	30 (6)	19 (7)
Bi/multilingual
No	389 (73)	193 (75)
Yes	144 (27)	66 (25)
Highest level of educationg
High school graduate	12 (2)	4 (2)
Some college or associate degree	75 (14)	38 (15)
College graduate	201 (38)	107 (41)
Some graduate school	46 (9)	19 (7)
Graduate, clinical, or professional degree	175 (33)	91 (35)

aFisher exact tests were used to examine differences in demographic characteristics between those who completed the training requirements and those who enrolled but did not complete the training.

bSample limited to learners who completed pre- and posttraining assessments.

cThree individuals were manually enrolled by the training team and therefore did not complete a registration form.

dIndicates significant result at P < .01.

eIndicates a significant result at P < .05.

fIncludes transgender male, n = 1; gender queer/gender nonbinary/gender nonconforming, n = 5.

gFor the course enrollees, 24 learners participated in cohorts that did not collect educational attainment data.

Pre- and postcourse knowledge assessment

Significant improvements from pre- to posttraining were demonstrated in 9 of 11 knowledge domains (P < .05) (Table 2). In addition, there was a significant increase (15.6%; from 8.63 to 9.98) in the composite knowledge score from pre- to posttraining (P < .001).

TABLE 2

Change in Knowledge of School Specialist Training Learners From Pretraining to Posttraining (N = 262) for Learners Who Completed Pre- and Posttraining Evaluations From February 8 to May 18, 2021

Individual Knowledge Domainsa,b	Pre (n = 262), % Correct	Post (n = 262), % Correct	P
Q1: Scope of School Specialist role	89%	97%	<.001c
Q2: Organization of COVID-19 response in schools by local health jurisdictions	94%	99%	.002c
Q3: Epidemiology and clinical manifestation of COVID-19 in children	73%	86%	<.001c
Q4: Mechanisms underlying reduced susceptibility to COVID-19 among children	81%	99%	<.001c
Q5: School reopening in relation to county tier level	28%	69%	<.001c
Q6: Steps to effectively de-escalate emotion	84%	94%	<.001c
Q7: Quarantine duration for asymptomatic close contacts	70%	86%	<.001c
Q8: Definition of a line list for investigating COVID-19 cases in schools	82%	96%	<.001c
Q9: Method of access to SPOT	81%	81%	1.00
Q10: Legal actions to prevent the spread of communicable disease available to local health officers	86%	94%	.001c
Q11: School outbreak reporting mandates required by California Assembly Bill (AB) 685	95%	98%	.06
Composite knowledge scored	8.63	9.98	<.001c

aRefer to Supplemental Digital Content Appendix A (available at http://links.lww.com/JPHMP/A951) for full questions and answer choices included on the surveys.

bFor each individual knowledge question domain, the McNemar test was used to assess differences between pre- and posttraining scores.

cP < .01.

dFor the composite knowledge score, the Wilcoxon signed-rank test was used to assess differences between pre- and posttraining scores.

Pre- and postcourse confidence in skills assessment

There was a significant improvement in self-perceived confidence in all 6 skill self-assessment questions from pre- to posttraining (Table 3). In addition, there was a significant improvement (20.1%; from 22.2 to 26.7) in the composite confidence score from pre- to posttraining (P < .001).

TABLE 3

Change in Confidence to Perform Skills Central to the Role of a School Specialist for Learners Who Completed Pre- and Posttraining Evaluations From February 8 to May 18, 2021

	Pretraining (n = 262), Mean (SD)	Posttraining (n = 262), Mean (SD)	P	Postdeployment (n = 27), Mean (SD)
School Specialist skill domainsa,b
Encourage people to share accurate information about their activities	3.89 (0.984)	4.41 (0.742)	<.001c	4.48 (0.700)
De-escalate emotion when engaging with an overwhelmed client	3.93 (0.968)	4.39 (0.733)	<.001c	4.48 (0.700)
Interpret data in a line list of cases	3.66 (1.24)	4.57 (0.717)	<.001c	4.48 (0.802)
Deduce important questions to ask of a school liaison	3.62 (1.14)	4.44 (0.764)	<.001c	4.41 (0.797)
Understand data sharing regulations to protect student and staff information	3.58 (1.22)	4.49 (0.736)	<.001c	4.58 (0.643)d
Perform School Specialist duties	3.48 (1.22)	4.38 (0.778)	<.001c	4.59 (0.636)
Composite confidence scoree	22.2 (5.74)	26.7 (3.97)	<.001c	26.9 (3.62)

aRefer to Supplemental Digital Content Appendix A (available at http://links.lww.com/JPHMP/A951) for full questions and answer choices included on the surveys.

bWilcoxon signed-rank test will be used to assess significant differences in confidence in skills from pre- to posttraining.

cP < .01.

dMissing response from 1 individual.

eWilcoxon signed-rank test will be used to assess significant differences in composite confidence score from pre- to posttraining.

Perceived preparedness for activation

Among the 60 field survey respondents, 27 (45%) reported having been deployed to work as School Specialists. Trainees' confidence postactivation from the field survey results (n = 27) was either maintained or slightly increased compared with the pre/postcourse assessment (n = 262) for all skills except the ability to interpret data in a line list of cases (Table 3). In addition, across all 6 skills, the average score was above a 4, which indicates trainees felt fairly confident in their ability to perform the skills.

Table 4 presents the learning areas trainees felt they needed additional knowledge and/or skills. For additional knowledge, trainees identified legal aspects of working with schools and minors as the area with the highest need, whereas case investigation/contact tracing was identified as the area with the lowest need. For additional practice in skills, trainees identified outbreak management as the area with the highest need, whereas case investigation/contact tracing and COVID-19 prevention in school settings were identified as the areas with the lowest need.

TABLE 4

Perceived Need for Additional Knowledge and/or Practice (N = 27) of School Specialist Training Learners Who Were Activated to the School Specialist Role From February 8 to May 18, 2021, Trainings

School Specialist Duties	Need More Knowledge, n (%)	Need More Practice, n (%)
Data management	7 (26)	8 (30)
Case investigation/contact tracing	2 (7)	4 (15)
Outbreak management	4 (15)	11 (41)
CalCONNECT's SPOT portal	7 (26)	9 (33)
School line list	4 (15)	5 (19)
LHJ's decision tree(s)	6 (22)	8 (30)
COVID-19 prevention in school settings	4 (15)	4 (15)
Legal aspects of working with schools and minors	11 (41)	7 (26)
Health coaching	5 (19)	7 (26)
Supporting schools with school reopening plans	7 (26)	9 (33)

Abbreviations: LHJ, local health jurisdiction; SPOT, Shared Portal for Outbreak Tracking.

Discussion and Conclusion

The School Specialist training improved trainee knowledge in key content areas needed for the School Specialist role. Trainees reported a higher level of confidence to perform School Specialist duties after completing the training. This evaluation demonstrated the course is effective at training LHD staff and state redirected staff in acquiring the necessary knowledge and confidence before School Specialist activation.

Throughout the courses, evaluation data were reviewed regularly for constructive feedback and updates were made to the training curriculum based on the findings. For example, instruction time was increased or the instruction was modified when a lack of improvement was seen such as for data management. The 2 knowledge questions where learners did not experience significant improvement pre- and postcourse completion were related to SPOT, the digital exposure reporting tool for school exposures, and the laws on reporting requirements for known or suspected COVID-19 cases in schools. This finding was not surprising, given the trainees had not yet been activated as School Specialists. Even for trainees with prior public health experience, this may have been their first time working on public health in schools, given the School Specialist role was a new LHD cadre of workers developed for California specifically for the COVID-19 response. In addition, SPOT was developed for COVID-19 and therefore trainees had never used this tool before. The incorporation of a demonstration of this tool in the curricula was not meant to be a robust training but rather to advertise its availability and to encourage trainees to sign up for a more in-depth training after the course.

Trainees showed improvement in confidence for all of the skill domains after course completion, demonstrating the strength of the hands-on, facilitator-led skill development activities that took place in virtual breakout rooms during the training. Some of the domains, such as the ability to encourage people to share accurate information, the ability to perform School Specialist duties, and understand data sharing regulations, showed even higher improvement after learners were activated in the School Specialist role. Trainees likely solidified their learning when putting their knowledge and skills into practice while on the job. A small number of skill domains showed a decrease after trainees activated as School Specialists, including the ability to interpret data in a spreadsheet and the ability to deduce important questions to ask school liaisons. Overall, the positive findings support a continuation and scale-up of the School Specialist training course. As of January 2022, the VTA+ team was continuing to expand, offering the course to new LHDs across the state of California.

Trainees who completed the field survey reported needing more support to understand the legal aspects of case and exposure reporting, data management, and the SPOT digital exposure reporting tool, and supporting schools with reopening plans. These findings align with the low improvement in knowledge in these content areas observed during the pre- and postcourse assessments. In addition, trainees reported needing the most practice in outbreak management, digital exposure reporting (SPOT), supporting schools with reopening plans, and using decision trees. Outbreak management was not an area extensively covered in the School Specialist course, but the VTA does have an in-depth course that focuses on how to prevent, respond, and manage outbreaks. This highlights the potential for integrating the VTA's School Specialist and outbreak management trainings, referring trainees to take both courses when necessary, or strengthening the current course in the domains of the legal aspects, data management, and outbreak management. In particular, outbreak management in K-12 schools will become more pertinent in future academic years with students returning to in-person learning and the emergence of more contagious variants of the COVID-19 virus such as Omicron.

There are several limitations with the School Specialist training evaluation. While the VTA+ training team interviewed and held focus group discussions with LHD staff to inform curriculum development, the VTA+ training team did not meet directly with school officials when developing the course. The trainees did not represent diverse racial and ethnic backgrounds, which is critical, given the equity implications of reopening schools and the significant toll COVID-19 has had on people of color. Trainees of Hispanic, Latina/Latino/Latinx, or Spanish origin were particularly underrepresented at 26% of trainees compared with their representation in the population of California at 39%.14 Surveys were administered online so that trainees could refer to training materials while taking the postcourse assessment. Trainees completed the pre- and postcourse assessment before being activated as School Specialists; therefore, the knowledge and skill assessments serve a proxy for how prepared trainees would be for this type of position. It is important to note that only 27 trainees were included in the postactivation field survey analysis. The findings from the field survey cannot be generalized to the larger group of learners due to the low response rate of 17%. We were not able to determine what proportion of trainees were activated as School Specialists by the LHDs. Trainees who were state redirected staff had not always been assigned to an LHD prior to course completion, meaning it was not confirmed they would be activated as School Specialists after completing the course. In addition, we excluded the 4 cohorts prior to February 18, 2020, from the analysis, given they did not complete pretraining assessments, meaning we missed an opportunity to assess 140 additional trainees who may have been activated in the School Specialist role.

Despite these limitations, this evaluation study found that the California VTA School Specialist training course was successful in improving trainees' knowledge and skills in key content areas that enabled them to serve the School Specialist role at LHDs.

It is critical that all children can receive in-person learning in a safe school environment,15 and it is an issue of equity, as the most disadvantaged children are most affected by school closures and quarantine.

LHDs need the capacity to support schools as they reopen and stay open amid the ongoing pandemic.

The lessons learned from the VTA School Specialist course can help inform similar health department programs in other states to train staff in a similar school support position.

Lessons can also apply to public health and educational collaborations for other diseases such as tuberculosis or other health promotion topics relevant to K-12 students such as obesity prevention or vaccination campaigns.