What Happens When Training Goes Virtual? Adapting Training and Technical Assistance for the School Mental Health Workforce in Response to COVID-19.

The Northwest Mental Health Technology Transfer Center (MHTTC) provides workforce training and technical assistance (TA) to support evidence-based school mental health practices. Historically, this support targeted school professionals through in-person and online trainings, workshops, and coaching. However, in response to COVID-19 restrictions, all support moved to online formats, and the Center introduced trainings for families and caregivers. The purpose of this article is to present preliminary process and outcome data that compare the reach and impact of support before and following COVID-19-related restrictions. Results suggest that transition to online support resulted in a wider reach and a more diverse audience, with no decrease in trainee satisfaction and perceived impact. Furthermore, families and caregivers reported positive gains in knowledge and behaviors following participation in a virtual youth suicide prevention training. Together, these findings suggest that online training and TA can provide tangible benefits to professionals and family members who support student mental health.

Introduction

Successful implementation and sustainment of strategies to support school mental health (SMH) require a skilled team of teachers, administrators, counselors, and other specialists (Damschroder et al., 2009; Fixsen et al., 2005; 2009; Le et al., 2016; Wandersman et al., 2012). The federally funded Northwest Mental Health Technology Transfer Center (MHTTC) provides workforce training and technical assistance (TA) to support the implementation, sustainment, and evaluation of evidence-based mental health practices to four states (Alaska, Idaho, Oregon, and Washington) and includes a team that specializes in providing such support for successful implementation of mental health practices in schools. However, impacts of the global coronavirus pandemic—including social distancing guidelines, shifts in needs of students, parents, and the school workforce, and subsequent changes in priorities of school leaders—have led to necessary adaptation of training and TA. Specific changes have included shifting to online formats for training and TA, holding entire conferences virtually, and, in light of school closures and limited access to school-based mental health services and supports, provision of support directly to parents and caregivers.

The purpose of the research described in this article was to examine process and preliminary outcome data focused on COVID-19-related adaptations made to SMH trainings and TA offered through the Northwest MHTTC. In the next section, we describe how lessons learned from implementation science theory along with assessments of current SMH needs have framed the SMH-related work of the Northwest MHTTC and its subsequent responses to COVID-related restrictions.

Using Implementation Theory to Reach the SMH Workforce

The training and TA activities supported through the Northwest MHTTC are grounded in the Consolidated Framework for Implementation Research (CFIR; Damschroder et al., 2009). The CFIR has been described as “meta-theoretical” as it draws from multiple theories of implementation and provides an overarching framework within which more explanatory theories can fit. As such, the model provides an excellent frame through which to understand the multiple, interrelated factors that influence implementation processes, and to identify and target areas in which training and TA can facilitate the uptake of evidence-based practices (Damschroder et al., 2009; Damschroder & Hagedorn, 2011). In the following paragraphs, we provide a brief overview of the five major domains of the CFIR and describe how the Northwest MHTTC has drawn from these domains to guide the development and delivery of SMH-focused trainings and TA.

Characteristics of Interventions

The CFIR underscores the importance of considering how characteristics of interventions themselves influence implementation outcomes. Interventions come from various sources, and are characterized by differing degrees of quality, adaptability, and complexity. Interventions that are appropriate for some populations may not fit within school settings or may not exhibit evidence of efficacy or effectiveness for school-aged youth. Other considerations that might affect acceptance of an intervention include costs, fit with current needs, and subjective perceptions of a particular intervention among school teachers, staff, and administrators (Bertram et al., 2011; Damschroder et al., 2009; Damschroder & Hagedorn, 2011).

In light of the important influence of intervention characteristics on implementation outcomes, the Northwest MHTTC employs a deliberate process to identify topics for trainings and TA. To date, content related to specific interventions has been informed by needs assessment data and state and school district policies and procedures. These efforts have resulted in training and TA content that is aligned with the unique needs of school districts throughout the northwest region and current professional development standards for education systems. Training and TA topics have focused on evidence-based and evidence-informed interventions implemented within a multi-tiered system of support (MTSS). Some examples include positive behavioral interventions and support (PBIS), universal behavior screening, dialectical behavior therapy (DBT) in schools, recognition and referral training to prevent suicide, school discipline, functional behavior assessments and behavior intervention plans, cognitive behavioral intervention for trauma in schools (CBITS), and psychological first aid.

Characteristics of Individuals

Interventions are implemented by individuals, and these individuals make choices that influence decision making related to evidence-based practices. Individual attributes such as knowledge of an intervention, self-efficacy, motivation, values, competence, and learning styles all have significant influences on the degree to which an intervention is accepted and ultimately implemented within a school or other community setting (Damschroder et al., 2009). To address individual-level influences on implementation processes, the Northwest MHTTC tailors training content by building relationships with school administrators, teachers, and other SMH personnel.

Since its inception, the Northwest MHTTC has built on existing relationships with intermediaries such as state and regional centers of excellence, state principals’ associations, and state education agencies. Over time, the team has expanded its network by reaching out to other entities that work directly with districts and schools, such as the Association of Washington School Principals, Northwest Positive Behavioral Interventions and Supports (PBIS) Network, the Special Education Services Agency in Alaska, and the Center for the Improvement of Student Learning at the Washington Office of Superintendent of Public Instruction. The team has also engaged with school and district staff at conferences and extended its reach through e-mail listservs.

These efforts have provided a means for the MHTTC team to understand the needs of education and community mental health professionals in districts and schools and develop and/or tailor training and TA to meet the needs of the SMH workforce. Training and TA delivery are based on a tiered model ranging from universal (“Tier 1”) to targeted (“Tier 2”) to intensive (“Tier 3”). Prior to COVID-19 adaptations, universal trainings included offerings for any interested stakeholders, typically in the form of in-person seminars, conferences, and occasional online webinars that were used to supplement and broaden the reach of training activities. Duration of didactic Tier 1 trainings generally ranged from 1 to 3 h, and the Northwest MHTTC also has distributed an electronic newsletter to a mailing list comprising school and community mental health professionals identified through attendance lists of MHTTC-sponsored events and related SMH conferences and mailing lists.

Tier 2 supports are more targeted than the universal efforts and have been offered as focused workshops, seminars, and communities of practice for smaller groups of individuals, schools, or districts. Prior to the onset of COVID-19, targeted efforts typically followed a similar format as for Tier 1, but included more personalized content tailored to individual participant needs. Finally, the Northwest MHTTC has provided intensive Tier 3 support for a few school districts each year through ongoing TA, consultation, and coaching, including long-term follow-up support ranging from months to a year or more. Across the three tiers of support, the Northwest MHTTC has implemented more than 40 in-person training sessions and TA efforts per year. These efforts have been geographically distributed throughout the northwest region, although suburban areas were overrepresented during the first 2 years of the grant.

Implementation Process

A third domain of the CFIR focuses on the specific processes by which interventions are implemented. These processes typically include planning, engaging appropriate individuals, executing implementation plans, and evaluating processes and outcomes (Damschroder et al., 2009). The Northwest MHTTC supports such implementation processes within most of its training efforts, and implementation support is a primary focus of intensive TA efforts. For example, the Northwest Interconnected Systems Framework (ISF) demonstration site project is a Tier 3 effort that provides intensive ongoing coaching and TA support to three Northwest school districts (OSEP Technical Assistance Center on Positive Behavioral Interventions and Supports, 2019). Such coaching efforts specifically focus on helping SMH professionals initiate and maintain processes to implement new SMH strategies, track progress, and evaluate outcomes.

Focusing on the Inner Setting

A fourth domain of the CFIR focuses on the contextual factors within organizations that influence the uptake of specific interventions (Damschroder et al., 2009). Examples of inner setting predictors of implementation include an organizational climate that is supportive of implementation and offers opportunities for training and workforce development, supportive leadership, and adequate resources (Aarons et al., 2011; Damschroder et al., 2009; Damschroder & Hagedorn, 2011; Moullin et al., 2019). Inner settings are important to consider when planning training, TA, and other forms of support since even the best-designed efforts will likely fail if participants return to settings that are unsupportive of implementation.

The Northwest MHTTC has specifically included trainings designed to promote organization-level environments that are conducive to implementing and sustaining high-quality SMH programs, policies, and interventions. Many schools express strong interest in providing a range of mental health supports to students, but struggle to effectively build their capacity. Leadership is critical for any organizational practice change, but school leaders (i.e., principals) are often inadequately equipped to fully support mental health programming within their buildings and districts (Lyon & Bruns, 2019; Smith-Millman and Flaspohler, 2019).

To address such needs, one of the goals of the Northwest MHTTC has been to support educational leaders to promote mental wellness for all (i.e., students, staff, families) through professional development to enhance strategic mental health leadership in schools. In Washington State, for example, the Northwest MHTTC partnered with the Association of Washington School Principals to co-develop and co-sponsor several training and TA events to increase competence and confidence of school leaders to integrate SMH within an MTSS framework. Similarly, as noted earlier, the center has provided training, coaching, and professional learning communities on the ISF. Both the MTSS and ISF models include specific readiness steps for schools and districts around establishing leadership teams, developing workplans, using data to make decisions, and deploying accountability measures focused on developing effective SMH supports. MHTTC trainings have also focused on adult wellness strategies and systems to prioritize self-care among staff. Together, these efforts are aimed at building positive inner settings that are supportive of implementation efforts.

Importance of Outer Settings

Although attending to inner settings is necessary, the CFIR model suggests that doing so is not sufficient to ensure implementation success (Damschroder et al., 2009; Watson et al., 2018). This results from the fact that inner settings are embedded within and influenced by outer settings, which include economic, political, and social contexts that shape opportunities and constraints for implementation (Bond, 2018; So et al., 2019; Williams et al., 2018). For example, school districts are influenced by state- and local-level policies, economic factors, and changes in administrative priorities. Each of these factors influences the degree to which schools have the resources and capacity to implement strategies that support the health and well-being of students.

Recently, the Northwest MHTTC, like other organizations, has been forced to adapt to one of the most influential outer setting events in recent history—the onset of the COVID-19 global pandemic. From an implementation science lens, the onset of the pandemic has created a unique opportunity to examine the impact of an unusually rapid shift in outer settings that has had a profound influence on SMH systems. In contrast to the typical slow, incremental nature of the policymaking process, the global pandemic has demanded a rapid policy response at federal, state, and local levels (Baumgartner et al., 2018; Kingdon, 1995). These outer setting changes have in turn had a significant impact on the inner settings of education systems throughout the USA. For example, school districts have shifted to online learning systems, and in-person support services for students have been severely restricted.

School teachers, administrators, policymakers, and researchers have acknowledged the multiple challenges that recent policy changes have placed on supporting the mental health needs of school-aged young people (Golberstein et al., 2020; Lee, 2020). In a textbook example of the intersection of inner and outer settings, the rapid onset of the COVID-19 pandemic has created a paradox in which the need for mental health support has increased and changed in scope in response to the stress and uncertainty stemming from the spread of the virus. At the same time, new policies have restricted access to the very services and supports necessary to respond to existing and emerging mental health needs (Golberstein et al., 2020; Lee, 2020).

Since the onset of COVID-19-related restrictions and policy changes, the Northwest MHTTC has adapted its entire approach to training and TA in ways designed to maximize its reach and relevance in response to the changing SMH landscape. As of March 26, 2020, when the Washington state (home state of the Northwest MHTTC) governor issued stay-at-home orders, all recent MHTTC support has been adapted to be delivered virtually. As a result, in-person events and conferences were canceled and the duration of most Tier 1 and 2 trainings was reduced. The Center pivoted to deliver a larger number of 60–90-min webinars and deliver Tier 3 support virtually. In addition, in response to school closures, decreases in access to mental health services and supports, and academic, economic, and emotional stresses on youth and caregivers (Golberstein et al., 2020; Lee, 2020), the MHTTC began offering trainings directly to families and caregivers of school-aged youth. One example is the LEARN® Saves Lives (LSL) program, a suicide prevention effort that targets parents and caregivers of school-aged youth.

The Current Study

Throughout the COVID-19 pandemic, the MHTTC has strived to inform its work through evaluation of the impact of the modifications and adaptations described above. The current paper aims to present a summary of the results of this evaluation effort, including its contribution to our understanding of (1) the impact of moving to virtual training and TA and (2) the effectiveness of directly targeting families and caregivers.

Study 1: Reach and Impact of Online Training

The first study aimed to evaluate the reach and impact of adapting MHTTC trainings to online formats. Recent research has suggested that online education can impact reach in several important ways. First, trainings delivered through online platforms have the potential to remove barriers that prevent some professionals from attending in-person events, such as those in remote areas, or those employed by organizations with limited travel budgets (Mullin et al., 2016). However, online resources may also be inaccessible by some groups due to limited Internet resources and limited knowledge of how to use virtual platforms (Alobuia et al., 2020; Hooper et al., 2020; Yancy, 2020). These inequities in access to adequate online resources have been exacerbated by the move to remote work by many employers in response to the pandemic, where employees rely on their own equipment to access Internet-based resources (Hooper et al., 2020; Yancy, 2020). As such, a primary purpose of the current study was to evaluate (1) the impact of the move to online trainings on overall attendance; (2) demographic characteristics of participants compared to pre-COVID-19; and (3) the extent to which the move to online training impacted participant satisfaction and self-reported outcomes associated with participation.

Study 2: Impact of Directly Reaching Families and Caregivers

Recent studies have suggested that parental involvement is linked to child access to mental health resources and treatment, and suicide awareness among caregivers has been associated with reduced risk factors among youth (Hooven et al., 2012; Kelly et al., 2007; Radovic et al., 2015; Rickwood et al., 2005; Sayal, 2006). As such, we expect that by actively engaging families and caregivers, the LSL training will help not only fill a void caused by school closures, but also will have a positive impact on suicide prevention behaviors within home settings. Study 2 thus aimed to evaluate (1) the reach of the LSL training in terms of the demographics of participants; (2) quality of the training as rated by participants; and (3) whether participant suicide prevention practices changed following the training.

Methods

Study 1

Data to assess the first goal of this study were gathered from 1718 individuals who participated in one or more of the SMH trainings offered through the Northwest MHTTC between January and June 2020. All participants were asked to respond to a post-event survey that assessed demographics, perceptions of training quality, and intentions to use the information shared during the session. Data for in-person trainings were collected via paper-and-pencil surveys that were administered immediately following the conclusion of each session. For online trainings, data were collected using an online version of the same survey that was administered through REDCap immediately following the session. The overall response rate for trainings that occurred prior to COVID-19-related restrictions was 66.4%, and the response rate during restrictions was 43.4%. The reduced response rate is likely due to the shift to exclusive online training following COVID-19-related restrictions. Online surveys typically result in lower response rates than in-person surveys, and our response rates are in line with what has been observed in other studies (Daikeler et al., 2020; Manfreda et al., 2008).

Study 1 Measures

Demographics

Demographics for all participants were assessed immediately post-event using a SAMHSA-required survey that aligns with the Government Performance and Results Act (GPRA). Participants were asked to indicate gender, race, ethnicity, and education level.

Reach

The reach of trainings and online resources was assessed through several sources of data. First, attendance at training events was tracked through administrative records. Second, Web traffic on the MHTTC Web site was tracked through Unique Page Views (UPV), which aggregates page views by a user during a single browsing session. Third, we assessed the reach of videos by counting the number of YouTube views for each SMH video hosted by the Northwest MHTTC. Finally, we assessed the reach of newsletters by tracking open rates for seven newsletters sent before COVID-19 restrictions and seven newsletters sent during restrictions.

Impact of Training

The impact of each training was assessed immediately post-event using the Impact of Training and Technical Assistance (IOTTA) survey (Coldiron, Walker, & Hensley, 2015; Walker and Bruns, n.d.). The IOTTA has been used to assess training outcomes for a wide variety of events focused on supporting the implementation of behavioral health interventions. Internal consistency of its multi-item subscales has been acceptable (see coefficients below). The following constructs were measured using items from the IOTTA:

Participant Mastery

Separate single items were used to assess self-reported mastery of training content. Participants were asked to rate their level of mastery before participating in the training on an index ranging from “0” for “complete beginner” to “10” for “fully expert.” A second item using the same response categories was used to assess post-training mastery.

Training Quality

Two items were used to assess training quality at post-event. Participants were asked to rate level of training organization and the degree to which the training held their attention. Response categories ranged from a low of “0” to a high of “10.” Responses to the two questions were averaged to calculate a composite training quality score (r = .70)

Trainer Credibility

A single question asked participants to rate the degree to which the trainer was credible in terms of being fully competent and having a high level of expertise relevant to helping trainees achieve the training goals.

Impact on Practice

Training impact was assessed post-event using eight items from the IOTTA survey. Each item began with the statement, “Since the training, how have the following aspects of your work changed?” Example categories include “How you understand student needs” and “How you interact with students.” Response categories ranged from -3 for “large negative impact” to 3 for “large positive impact.” Scores on the individual items were averaged to form a single composite score (α = .92).

Study 1 Procedures

The Northwest MHTTC has implemented several procedures to extend its reach during the pandemic. First, existing webinar and virtual training capacity was increased by expanding attendance capacity. Immediately following COVID-19-related stay-at-home orders, the Northwest MHTTC shifted Tier 1 and 2 events to large capacity Zoom rooms (limit of 500) to accommodate an increase in the number of people who would need professional training and support. In addition, in light of increases in both the number and frequency of webinars, the Center sent additional e-mail blasts to remind people of upcoming events, and adapted planning timelines to allow for posting of webinars and other resources within a shortened 2–3-day time period. Tier 3 intensive training and TA support were moved to monthly Zoom meetings and e-mail discussions. School districts participants can bring challenges to regular check-ins for crowdsourcing of ideas. These regular meetings have established a learning community in which participants can lean on each other and adapt to pandemic-related challenges with the MHTTC providing opportunities for additional peer support for complex issues (Table 1).

Table 1

Changes in training format, target audience, and topics from pre-restrictions to during restrictions

	Pre-COVID-19 restrictions	During COVID-19 restrictions
Training formats (all tiers)	In-person seminars In-person workshops Regional conferences Communities of practice Webinars Newsletters	Virtual seminars and workshops using enhanced capacity Zoom rooms Webinars Newsletters
TA formats (Tier 3 only)	In-person meetings Telephone calls Individualized coaching	Monthly Zoom meetings Telephone- and Zoom-based coaching E-mail communications Online learning communities
Target audience	SMH workforce	SMH workforce Parents Caregivers Youth
Training topics	SMH-focused evidence-based practices (EBPs) Multi-tiered system of support (MTSS) Implementation processes Leadership support Workforce member well-being	SMH-focused EBPs MTSS Implementation processes Leadership support Workforce member well-being COVID-19 impact on SMH Telehealth best-practices Support for parents and caregivers

With the goal of matching training and TA content with emerging needs of the SMH workforce following the onset of the pandemic, the MHTTC training team developed a wide variety of new online resources, virtual events, and livestreamed trainings. These resources and trainings have been designed to not only address needs that existed prior to the onset of COVID-19, but also respond to new needs that have emerged as a result of the pandemic, and the anticipated needs educators will have when schools re-open. To assess these emerging needs, the Northwest MHTTC reached out to leaders of state education agencies, centers of excellence, established subject matter expert (SME) partners, and other professionals within local school districts to identify ways to expand the array of topics to meet the needs of the SMH workforce. In addition, a formal event was held early in the pandemic to gather school mental health practitioners to surface school mental telehealth needs and resources. Finally, evaluation reports of pre-COVID-19 trainings helped the team identify and prioritize specific topics and helped SMEs design and deliver high-quality professional trainings within virtual settings. Presenters modified content to address pandemic-related impacts on student mental health.

Study 1 Analysis Plan

The reach of MHTTC trainings before and after COVID-19-related restrictions was assessed through changes in simple descriptive data drawn from attendance records for trainings and Web analytics Web pages and newsletters. We examined numbers reached before and after the onset of restrictions and standardized these numbers as per-event participation rates. Web analytic data were summarized as percentage increases from pre- to post-onset of COVID restrictions in Web page visits and open rates of electronic resources. Demographic data were summarized as percentages to assess changes in audience characteristics across time periods. Changes in perceived quality of training events and intended impact of the events from pre-restrictions to post-onset of restrictions were assessed using simple independent samples t-tests, and Cohen’s d coefficients were calculated to estimate effect sizes. All analyses were run using SPSS version 27 with pairwise deletion of missing data.

Study 2

Data to assess the second goal of this study were drawn from pre-event, post-event, and follow-up online surveys of 309 parents and caregivers who participated in the LSL suicide prevention training program. These online surveys were administered through REDCap. The pre-test survey, which was administered the day before the online event, assessed demographics and included questions related to suicide prevention actions (N = 244, response rate = 79%). The post-event survey was administered at the end of the webinar and repeated the suicide prevention questions from the pre-test (N = 173, response rate = 56%; N = 129 paired responses from pre-test through post-test). The post-event survey also assessed perceptions of the quality of the webinar. The follow-up survey was administered 2 weeks after the webinar and included the same suicide prevention questions that were included in the pre-test (N = 55, response rate = 17.8%; N = 37 paired responses from pre-test through follow-up).

Study 2 Measures

Race, ethnicity, gender, and level of education were assessed with the GPRA survey at both pre-test and post-test.

Quality of Training

At post-event, participants were asked two questions to rate the degree to which the training held their attention and the degree to which it was well organized. Ratings were on an 11-point Likert-type index with response categories ranging from 0 for “not at all”/“completely disorganized” to 10 for “superbly organized/exceptionally well.” Scores for both items were averaged to calculate a two-item composite indicator of quality (r = .61)..

Knowledge

Participants were asked to provide ratings of their knowledge of three aspects of suicide prevention at all three measurement time points. Questions specifically asked participants to rate their level of competence in making their homes safer, asking their children about suicide in a safe way, and recognizing signs of a mental health crisis. Response categories ranged from 0 for “complete beginner” to 10 for “fully expert.” Three separate composite knowledge scores were calculated by averaging scores for the three items at each measurement time point. This knowledge index demonstrated acceptable internal consistency at all three time points (αpre-test = .93; αpost-event = .88; αfollow-up = .91)..

Suicide Prevention Actions

Participants were asked to respond to ten questions focused on specific actions that focus on suicide prevention at post-event and follow-up. Example items include: “Do you currently have the suicide prevention lifeline or crisis text chat as a contact in your mobile phone?” “Do you currently own a lock box to store medication?” “If someone was in mental health crisis in your home, would you be willing to limit their access to firearms?” All questions included “yes,” “no,” and “I don’t know” response categories. Items related to firearm safety were excluded from analyses due to low rates of firearm ownership among participants.

Study 2 Procedures

With the goal of directly reaching families and caregivers, the Northwest MHTTC adapted the LSL program to be delivered online. LSL training was provided in collaboration with its developers, Forefront Suicide Prevention. LSL training attendees were recruited by Forefront through its wide-reaching ListServ and social media networks, as well as its network of over 50 Washington State schools that participate in its Forefront in the Schools program. The LSL training provides information on essential skills for suicide prevention, including how to make a home safer to prevent the risk of suicide; how to ask children and youth about suicide in a safe way; and how to recognize signs of a mental health crisis and take practical steps to address it (Forefront Suicide Prevention, 2020). Although this training is typically delivered in person, the Northwest MHTTC partnered with Forefront Suicide Prevention to adapt it to a virtual format and tailor it to parents and caregivers as the primary audience.

Study 2 Analysis Plan

Ratings of perceived quality of the LSL suicide prevention training were summarized with simple descriptive statistics. Changes in self-reported knowledge scores were assessed with paired t-tests that compared pre-event scores with post-event scores. We also assessed the degree to which any gains in knowledge were maintained over time by using paired t-tests to compare pre-event and follow-up scores. Cohen’s d coefficients were calculated to estimate effect sizes. Finally, changes in suicide prevention practices were assessed by comparing percentages of participants following such practices at pre-event with the percentage following such practices at follow-up. Chi-square analyses were used to assess statistic significant of observed changes. All analyses were run using SPSS version 27 with pairwise deletion of missing data.

Results

Reach of MHTTC Resources Participation rates and Web site statistics demonstrate significant shifts in Northwest MHTTC training methods. During the pre-COVID restriction time period, the Northwest MHTTC provided 28 trainings to 1254 individuals, three of which were delivered online. In contrast, all 14 events held during the restrictions time period were delivered online to 2023 participants. These data suggest an increase from 45 participants per event during the pre-restrictions time frame to 145, post-restrictions. Demographic data suggest shifts in audience characteristics, with post-COVID restriction events reaching more female participants (80.5% pre; 89.4% post); more representatives of non-White racial/ethnic minority groups (10.9% pre; 19.4% post); and slightly more participants with a bachelor’s degree or less (31.3% pre; 37.7% post; see Table 2 for additional demographic data).

Table 2

Reach of Northwest Mental Health Technology Transfer Center (MHTTC) resources (N = 1672)

	Pre-restrictions (%)	During restrictions (%)
Gender
Female	80.5	89.4
Male	19.3	9.6
None of these	0.1	1.0
Race
American Indian/Alaska Native	3.4	2.8
Asian	1.8	4.2
Black or African-American	3.0	5.8
Hispanic or Latino	5.5	11.5
Native Hawaiian or other Pacific Islander	0.7	0.6
White	89.1	80.6
Degree
High school diploma or equivalent (GED)	0.5	1.2
Some college, but no degree	2.8	6.2
Associate’s degree	1.2	5.6
Bachelor’s degree	24.0	22.6
Master’s degree	65.6	56.5
Other doctoral degree or equivalent	3.1	5.8
Other	2.8	2.0

Web analytics for online resources indicate similar shifts in usage between pre- and post-onset of COVID restrictions. In response to the pandemic, the Northwest MHTTC posted 94 new SMH-related pages to its Web site, an increase of 196%. Web traffic across all SMH pages, as measured by unique page views (UPV), increased by 830% when compared to the pre-COVID restrictions time period (an increase of 23,713 views). Newly posted pages accounted for most of the new UPV (90.3% or 21,420 views); however, existing pages from the pre-COVID period also saw an 80.3% increase in UPV following the onset of COVID-19 restrictions. Most of the page views originated within the northwest region, although new UPV from other regions increased by 461.2%, suggesting additional reach across the USA.

In addition to increases in Web page views, the Northwest MHTTC posted 31 new SMH YouTube videos, an increase of 517%. The number of views across all SMH videos increased by 3927% and average watch time increased by 1126%. However, 38% of the new views and 58% of the additional watch time were accounted for by a single organization that engaged heavily with the Northwest MHTTC resources during the post-onset of COVID restrictions time frame.

Finally, data related to SMH newsletters suggest increased usage from pre-COVID restrictions to post-onset of COVID restrictions. Open rates for the five online newsletters posted pre-restrictions ranged from 11.2% to 41.7%, with an average open rate of 18.5%. In contrast, open rates for the five online newsletters posted after the onset of restrictions ranged from 20.3% to 44.6%, with an average open rate of 35%.

Perceived Quality of MHTTC Resources Despite the above-mentioned format and demographic changes, perceptions of the quality of events increased from the pre-restrictions to post-restrictions onset time period, demonstrating a moderate effect size (pre-restrictions = 8.35, post-restrictions = 9.03, t = − 10.06, p < .001, d = 0.50). In addition, ratings of trainer credibility increased slightly but significantly across time periods, demonstrating a small effect size (pre-restrictions = 8.82, post-restrictions = 9.18, t = − 5.65, p < .001, d = 0.28; Table 3).

Table 3

Changes in perceived quality and impact of pre- and during COVID-19 restrictions events

	Pre-restrictions		During restrictions		t test	Cohen’s d
	Mean	SD	Mean	SD
Quality	8.35	1.61	9.03	1.06	− 10.06**	0.50
Trainer credibility	8.82	1.47	9.18	0.96	− 5.65**	0.28
Baseline mastery	5.67	1.98	5.77	2.14	− 0.86	0.05
Post-event mastery	7.02	1.66	7.42	1.64	− 4.64**	0.24
Intentions to use	1.56	0.84	1.46	0.84	2.34*	0.08

*p ≤ .05, **p ≤ .01

Impact of MHTTC Resources Despite similar self-ratings in baseline mastery of training content prior to the event at both time periods (pre-restrictions = 5.67, post-restrictions = 5.77, t = − 0.86, p = .386, d = 0.05), self-reported post-event mastery was slightly but statistically significantly higher post-onset of restrictions relative to pre-restrictions (pre-restrictions = 7.02, post-restrictions = 7.42, t = − 4.64, p < .001, d = 0.24). However, intentions to use knowledge and skills gained from the trainings decreased slightly, but statistically significantly, across time periods (pre-restrictions = 1.56, post-restrictions = 1.46, t = 2.34, p = .019, d = 0.08; Table 3).

Reach of LSL Training As noted above, the LSL suicide prevention training was unique in that it targeted families and caregivers instead of SMH professionals. The event attracted 309 participants. Of the 244 who responded to the pre-event survey, 10.3% had a young child (age 0–4 years) living with them, 34.4% had a school-aged child (age 5–18 years) at home, and 20.5% had a young adult (age 19–24 years) at home. Consistent with other Northwest MHTTC trainings, LSL participants identified primarily as female (89.9%); most identified as non-Hispanic (88.1%) and White (74.6%); and most were highly educated (82.7% with bachelor’s or graduate degree).

Perceived Quality of LSL Training Data for the LSL suicide prevention event that targeted parents and caregivers suggest that the training was well received. Most participants agreed that they were satisfied with the event (99.4%); expected benefits from the session (98.3%); and would recommend the training to others (99.4%).

Impact of LSL Training Outcome comparisons between pre-test and follow-up suggest that LSL participants made measurable gains in response to the training. Paired t-tests indicate statistically significant increases in knowledge from pre- to post-test, demonstrating a large effect size (pre-test = 5.71, post-test = 8.31, t = − 15.82, p < .001, d = 1.39). Furthermore, these gains were maintained at a two-week follow-up assessment (pre-test = 5.71, follow-up = 8.19, t = − 7.83, p < .001, d = 1.29; Table 4).

Table 4

Changes in knowledge scores following participation in LEARN® Saves Lives (LSL) suicide prevention training

	Mean1 (SD)	Mean2 (SD)	t	Cohen’s d
Knowledge score pre- to post-test (n = 129)	5.71 (2.39)	8.31 (1.11)	15.82**	1.39
Knowledge score pre- to follow-up (n = 37)	5.41 (2.53)	8.19 (1.15)	7.83**	1.29

**p ≤ .01

Outcome data also suggest significant changes in several behaviors related to suicide prevention that were addressed in the training. Compared to pre-test, at follow-up, significantly more participants were likely to have a crisis number saved to a phone (35.1% vs. 62.2%, χ2 = 12.2, p < .001), own a lockbox for medication (30.0% vs. 40.0%, χ2 = 12.8, p < .001), and remove expired/unused medication (73.3% vs. 80.0%, χ2 = 12.3, p < .001; Fig. 1).

Fig. 1

Percent of LEARN® Saves Lives (LSL) suicide prevention training participants answering “Yes” to items related to suicide prevention (N = 37)

Discussion

The overarching goal of this study was to examine how shifts in training modalities adopted by the Northwest MHTTC in response to COVID-19 restrictions influenced the reach, quality, and impact of trainings targeted toward the SMH workforce as well as caregivers and family members. Taken together, current findings suggest that adapting trainings to virtual online formats and specifically targeting family members to be included in the training plans have yielded encouraging evidence of short-term impact. Specifically, participation increased dramatically in the months after lockdown, participants’ ratings of quality and impact did not decline, and provision of training directly to parents and caregivers, even via single events provided online, resulted in self-reported knowledge gains and behavior change. In the following paragraphs, we examine these findings further and interpret them within the context of the CFIR model and previous research on workforce development.

Reach of the Northwest MHTTC

The CFIR model underscores the important role that individual workforce members play in the implementation process (Damschroder et al., 2009), and data from Study 1 suggest that the center has increased its capacity to reach such individuals. First, overall attendance numbers, open rates, and video views suggest that the Northwest MHTTC is reaching more SMH professionals than prior to COVID-19 restrictions. Such data are consistent with previous research findings that online trainings have the potential to reach larger numbers of participants when compared to in-person trainings (Mullin et al., 2016).

Second, demographic data suggest that the shift to online training has increased participants’ diversity in terms of race/ethnicity and level of education. Specifically, trainings and TA provided during the COVID-19 restrictions time period have been attended and/or accessed by a larger proportion of professionals from non-White racial groups, and resources also have been accessed by individuals who are less likely to have advanced degrees. This finding is encouraging in light of emerging data from other studies suggesting racial/ethnic disparities in access to electronic digital formats for the dissemination of COVID-19-related knowledge (Alobuia et al., 2020; Hooper et al., 2020; Yancy, 2020). Indeed, the current data suggest that the shift to online training has made Northwest MHTTC resources more, rather than less, accessible to workforce members across racial and ethnic groups. In order to maintain or increase access across diverse groups over time, the Northwest MHTTC plans to engage communities of color, native communities and other minoritized educators, administrators and school mental health professionals through formal and in-formal needs assessments.

One area in which the Northwest MHTTC continues to struggle is in gender diversity. Before the onset of the pandemic, the Center trained and supported educators, administrators, and other professionals who mostly identify as female. This is not entirely surprising given that this breakdown of gender generally reflects the composition of the SMH workforce (Robiner, 2006). Following the move to online resources, however, the proportion of female participants increased even further. The reasons behind this shift are unclear—it is possible that more female members of the SMH workforce were moved to remote work situations and such arrangements enabled them to have better access to online resources. Whatever the cause, such data suggest a need for the Center to obtain an accurate assessment of the demographic composition of the SMH workforce and work to reach all members who need or desire training and/or support.

Quality and Impact of Trainings and TA

According to the CFIR model, characteristics of interventions and implementation processes also influence the uptake of evidence-based strategies. Thus, effective trainings and TA support must be tailored to help members of the SMH workforce gain knowledge and skills that will help them navigate complex implementation processes within school and community settings (Damschroder et al., 2009; Fixsen et al., 2005; Le et al., 2016; Wandersman et al., 2012). Results from the current study suggest that the Northwest MHTTC’s efforts to provide such training and support have been successful, with participants providing high ratings of quality and impact even after the shift to pure remote formats. In fact, participants reported higher mean ratings of event quality and trainer credibility following the move to exclusive use of online training. Furthermore, participants reported higher levels of content mastery following COVID-related changes, suggesting that participants have gained an understanding of intervention characteristics and implementation processes. Such findings are consistent with research that has suggested that well-designed and properly administered online trainings can be at least as effective as their in-person counterparts (Becker et al., 2014; Mullin et al., 2016).

It is possible that the outcomes observed in this study were influenced by the deliberate actions taken by the MHTTC to maintain relationships among trainers, TA providers, and participants despite (or because of) the move to online support. As noted in the introduction, these relationships play a pivotal role in identifying topics, establishing training and TA formats, and engaging with participants over time, and the Northwest MHTTC has taken concrete steps to maintain, and even strengthen these relationships. This may help explain positive outcomes since positive, supportive relationships with direct and clear communication are a key ingredient associated with implementation success (Le et al., 2016; Wandersman et al., 2012).

Interestingly, while ratings of event quality and personal mastery remained high, ratings of intentions to actually use the training material decreased slightly but statistically significantly following the move to online training. Such ratings may reflect participants acknowledging the reality of COVID-related constraints having a negative influence on their ability to actually implement strategies reviewed in the trainings, or they may reflect uncertainty about the future in light of the pandemic. This would be an example of outer setting constraints influencing inner setting conditions and thus limiting individual professionals’ abilities to effectively engage in implementation work (Damschroder et al., 2009; Damschroder & Hagedorn, 2011). Results may also reflect the increased number and diversity of education staff attending SMH-related events, including some staff persons with less explicit roles in implementing SMH strategies such as those covered in the trainings.

Lessons Learned from Extending Support to Families and Caregivers

Data from Study 2 indicate that the Northwest MHTTC has successfully implemented an online training that directly targeted parents and caregivers. Participants in the LSL suicide prevention training reported high levels of satisfaction. Moreover, preliminary outcome data suggest statistically significant knowledge gains and behavior changes related to home-based steps to prevent suicide. Such findings suggest that online SMH trainings tailored to families and caregivers may help address some of the need created by reduced student access to services and supports typically delivered directly through schools or other community organizations. Such community-focused training has been found to be important in the face of outer setting events that restrict usual routes of access (Jacobs et al., 2016).

Providing training and technical support directly to family members and caregivers may have additional benefits, such as providing support when there is a lack of resources for mental health, and/or underfunded SMH programs. Additionally, there are demonstrated direct benefits associated with involving families and caregivers into any efforts designed to support the mental health and well-being of young people. For example, previous research has suggested that teaching parents suicide awareness can reduce suicide risk factors among youth (Hooven et al., 2012). More broadly, even though research has indicated that parental involvement is key to getting children resources and treatment (Kelly et al., 2007; Radovic et al., 2015; Rickwood et al., 2005; Sayal, 2006), parents commonly cite a lack of knowledge of mental health problems as a barrier to accessing treatment (Crouch et al., 2019; Reardon et al., 2017). Empowering parents and caregivers by directly targeting them with training topics more typically targeted toward members of the SMH workforce may help increase parents’ and caregivers’ ability to advocate for their children and help them obtain appropriate services.

Finally, positive short-term outcomes found here provide encouragement about the Center’s ability to expand online SMH-focused training and support to individuals in other school roles. This is important given that researchers have stressed the importance of supporting a diverse array of individuals to implement strategies that can address student mental health challenges, rather than exclusively relying on mental health professionals. In addition to parents and caregivers, efficient modeling, coaching, and supervision of professionals working across a full range of roles can improve the overall capacity of schools to prevent and address student emotional and behavioral problems (Graczyk et al., 2003). Using CFIR language, training families and caregivers may help expand the number of individuals with the skills necessary to implement SMH strategies (Damschroder et al., 2009).

Limitations

As with any empirical project, several limitations limit the internal and external validity of this study. First, all data were self-report. As such, it is possible that the findings have been influenced by common response biases such as social desirability biases. Furthermore, although most variables were measured using established indices, several were assessed using single-item measures that are unlikely to capture the complexity of the underlying constructs.

Another limitation of this study is that while response rates for in-person events were acceptable, rates for online events were much lower. Furthermore, the response rate for the follow-up survey for the LSL suicide prevention training was very low. Low response rates have become common for online surveys, and the current rates are consistent (and in some cases surpass) commonly observed rates (Daikeler et al., 2020; Manfreda et al., 2008). However, the low rates suggest that the analytic samples for this study may not truly represent the population of MHTTC training participants. Thus, the conclusions from this study should be considered preliminary and caution should be exercised if applying these findings beyond the current sample.

A third limitation is that while there was some overlap in the training and TA topics before and following COVID-19-related restrictions, many topics were tailored to time-sensitive needs of participants. As such, it is possible that some of the differences in participant perceptions that we observed over time were due to reactions to topics rather than delivery formats. Future evaluations would benefit from data that differentiate between impact derived from training format versus impact derived from training content.

Finally, the methods employed to assess impact of the LSL training were limited by the lack of comparison group. Although the analysis of pre-, post-, and follow-up data provides evidence of changes over time, the lack of comparison group data limits our ability to definitively attribute any of the observed changes to the LSL training. For example, it is possible that training participants were already motivated to learn about suicide prevention and sought out additional resources that prompted the changes that were observed in this study. As such, any causal conclusions related to the impact of LSL should be considered preliminary, pending further investigations that include a comparison group.

Implications for Implementation Science

Despite these limitations, the current findings have implications for implementation theory and practice. The positive findings related to reach, participant satisfaction, and outcomes suggest that the specific steps taken to align the planning and delivery of training and TA activities with the five CFIR domains have been successful as practices were adapted to online platforms. The following guidelines may help other organizations facilitate effective adaptations of their own training and TA practices.

Intervention Characteristics Training content and delivery formats should be guided by careful consideration of how the characteristics and complexity of interventions might impact implementation, and TA providers should select training content based on the degree to which such complexity can be supported through online training and coaching.

Individual Characteristics Topics and formats should be tailored to intended audiences to ensure that the level of training and targeted skills fit with the particular needs of participants and that these needs can be addressed through online support. Furthermore, intensive online TA efforts should aim to establish and maintain regular communication and positive relationships among TA providers, SMH professionals, parents, caregivers, and students.

Implementation Processes Level of support offered through online trainings and ongoing TA should be tailored based on the resources participants have available to successfully navigate the processes of implementing SMH interventions. Particular attention should be given to intensive support efforts that typically require individualized TA and coaching.

Inner Settings Online trainings and TA should be designed such that new skills can be employed in light of the priorities, resources, and constraints that exist within participants’ organizations.

Outer Settings The current findings demonstrate that reach and impact of training and TA can be achieved in light of a rapid adaptation in response to disruptive outer setting events. Such findings underscore the importance—and capacity—of TA entities to pivot in response to other outer setting events such as policy shifts, changes in funding, or unexpected public crises.

Implications for School Mental Health

The results from both studies also have direct implications for SMH, suggesting that SMH trainings can be effectively delivered through online support. As suggested throughout this article, such findings are encouraging in light of the current COVID-related restrictions. However, online delivery also may have benefits that extend beyond the current pandemic. Specifically, our findings suggest that online trainings have the potential to reach a broader audience and engage individuals who would not otherwise access in-person events. As our social institutions emerge from COVID-19-related restrictions, there may be some benefit to continued use of online training and TA formats.

Implications for Family Outreach

Results related to the online LSL suicide prevention training provide further evidence that our responses to COVID-19-related restrictions may have benefits that transcend the current crisis. Indeed, involving families and caregivers in addressing the mental health needs of young people seems important at all times, not only during periods of crisis. The current data suggest that a simple training may help families and caregivers identify relevant resources to support their children and take specific steps to promote home safety and reduce suicide risk. Such efforts could be extended to increase parental and caregiver awareness of other signs of mental health need and empower them with the knowledge and skills to support their children. Engaging in such efforts alongside more commonly implemented strategies to support members of the SMH workforce may represent a positive development in mental health support that extends beyond the current global pandemic.

Conclusions and Future Directions

Together, the findings of the studies described in this manuscript suggest that in the face of a major outer setting event, training and TA for the implementation of evidence-based SMH interventions can be effectively delivered through online formats. Indeed, as a result of the move to virtual training, the Northwest MHTTC has broadened its reach, attracted a more diverse audience, directly reached families and caregivers, and maintained high levels of participant-reported satisfaction and impact. In the coming months, the MHTTC team will continue to refine and evaluate its efforts, incorporating new measures of impact and seeking to identify which training formats are associated with the most positive outcomes. We plan to specifically examine the impact of ongoing, intensive TA efforts to identify how best to support ongoing relationships among TA providers and TA recipients. Such information will be critical, especially given that, at the time of this writing, surges in COVID-19 rates suggest many more months of online training and TA. Looking beyond the current crisis, it is our hope that the lessons learned from our response to the pandemic will also have the capacity to inform virtual training and TA for SMH providers, students, and caregivers even after restrictions are lifted.