Evaluation of a Systems-Wide Telebehavioral Health Training Implementation in Response to COVID-19.

In response to the Coronavirus disease 2019 pandemic, it became important to rapidly train mental health providers to provide telebehavioral health services (TBH) within a pediatric care setting. This study examined the perceived usefulness of various TBH training materials; provider motivation, comfort, and confidence in implementation of TBH services; and perceived benefits and barriers of TBH. After completing various training options and implementing TBH services for 1 month, providers indicated all training materials were helpful, with the recorded webinar being identified as the most helpful resource. This study describes the rapid training of mental health providers in preparation of system-wide TBH services.

Introduction

Despite impressive technological advances, emergent research demonstrating the efficacy of telebehavioral health (TBH), widespread utilization of TBH interventions has been limited among mental healthcare professionals (1,2). Although some training resources are available, there is limited information on how to implement TBH on a large scale (3 –5).

The impact of the Coronavirus disease 2019 (COVID-19) pandemic in early 2020 led to a rapid transition to TBH for many mental health professionals and healthcare organizations (6,7). The present exploratory study reports on the experience of one institution as it trained providers in TBH due to the institution’s pivot to TBH in response to the COVID-19 pandemic. It was hypothesized that training materials would increase providers’ motivation to provide TBH, confidence in being able to competently screen patients for suitability for TBH services and in implementing TBH.

Methods

Design

All mental and behavioral health providers within a large Midwestern children’s hospital system were asked to participate in a pretraining survey. Providers were sent an e-mail with a link to a 6-item pretraining survey created by the study investigators. This pre-survey was completed by 87 providers and was used to inform specific training materials available to providers. Ratings were on a scale from 0 to 5, with 5 indicating highest degree of comfort, motivation, or confidence.

After the pretraining survey was completed, providers were able to self-select which TBH trainings they completed on their own time and in any location. No incentives were offered for completing the trainings. All providers were then asked to exclusively provide TBH services due to in-person restrictions related to the COVID-19 pandemic. After approximately 1 month of providing TBH, providers were asked to complete a post-implementation survey.

Training Description

Providers were able to access 4 sets of training materials created based on evidence-based practices and provider feedback.

Telebehavioral health training session

A prerecorded TBH training video (approximately 1 hour in length) was provided as a set of PowerPoint slides and audio that highlighted evidence-based research associated with TBH, considerations for practice, technology considerations, legal and ethical factors, determining appropriateness of services using TBH, crisis management planning, considerations in documentation, and factors to consider when developing a therapeutic virtual milieu.

Quick guide: TBH training

A quick guide to TBH training was presented as a set of PowerPoint slides that provided a more concise version of the more comprehensive aforementioned training with greater emphasis on the practical implementation of TBH services.

Additional clinical resources

Two written documents and a mock TBH video were made available to assist clinicians with their espoused concerns of managing psychiatric emergencies remotely, setting up their home office appropriately, and translating therapeutic approaches to a videoconferencing format.

Technical training resources

Providers also had access to additional technical resources including written handouts that assisted providers with how to schedule a video visit, how to connect to video and start a video visit through the hospital system, how to utilize video conferencing control settings, how to document verbal consent and assent for TBH treatment, how to perform basic technical troubleshooting during a video visit, and how to document appropriately for reimbursement of services using the hospital medical record system.

Surveys

After reviewing TBH training materials and starting TBH with patients for approximately 1 month, providers were asked to complete a 23-item post-implementation survey.

Data Analysis

One-sample t tests were conducted to analyze differences in mean provider ratings from pretraining to post-implementation surveys for 3 questions related to providers’ motivation to provide services, comfort screening patients for appropriateness of services, and overall confidence facilitating services over TBH. It was hypothesized that provider responses across all 3 questions would be significantly improved following training and implementation of TBH services.

Results

Participants

Data were gathered from 87 providers who completed the preimplementation survey in March 2020. In order to prioritize anonymity, no provider demographic information was obtained for this survey. A predetermined sample size had not been chosen since the focus for providing this training was to allow for any and all providers to self-select based upon their own interest. Follow-up data, including demographic information, were gathered from the 53 providers who completed the post-training survey in May 2020 (See Table 1).

Table 1.

Participant Demographic Information.a

	n (%)
Gender
Female	45(85)
Male	8(15)
Race/ethnicity
African-American/black	6(3)
White	47(89)
Latinx	1(1)
Other	1(1)
Age
20-30	8(15)
31-40	43(23)
41-50	26(14)
51-60	9(5)
61+	6(3)
Highest level of education
Master’s degree	23(43)
Doctoral degree (MD or PhD)	30(57)
Occupation/current position
Master’s level therapist	21(40)
Psychologist	26(49)
Psychiatrist	4(8)
Other	2(4)
Years of work in mental health
0-3	6(12)
3-6	16(31)
6-9	5(10)
9+	25(48)

a n = 53.

Pretraining Survey Results

Prior to offering training and completing TBH, 67% of providers that completed the survey indicated feeling very motivated or extremely motivated to offer TBH servicers with only 13% reporting prior experience with TBH. The 2 most frequently identified concerns about providing TBH were questions regarding the effectiveness of TBH (57%) and concern regarding how to adapt their therapeutic style for a virtual service delivery (49%). Approximately 43% of providers felt somewhat confident (43%) in being able to provide TBH prior to training.

Post-Implementation Survey Results

Information regarding participant engagement, perception of training materials, and desire for additional training can be found in Table 2. Following training and implementation of services, 83% of providers that completed the survey reported feeling very motivated or extremely motivated to continue offering TBH at the hospital. Nearly all providers (98%) reported feeling confident about facilitating TBH at the time of this latter survey.

Table 2.

Participant Post-TBH Training Engagement and Perceptions.a

	n (%)
Type of trainings completed
Recorded webinar	41 (89)
Quick guide	33 (72)
Mock therapy video	13 (28)
Technical resources	30 (65)
Training material rated as very/extremely helpful
Recorded webinar	30 (60)
Quick guide	23 (47)
Mock therapy video	12 (26)
Technical resources	18 (37)
All training materials provided	29 (60)
Additional training support desired
Live webinar	9 (20)
Additional recorded webinars	16 (36)
Consultation with TBH Expert	17 (38)
Consultation with IT Staff	14 (31)
Other	12 (27)

Abbreviation: TBH, telebehavioral health.

a n = 53.

After completing their preferred training materials, the majority of providers (64%) indicated that they had conducted at least 21 TBH visits over the past month. All providers indicated that they perceived TBH sessions as being beneficial with the greatest benefit observed in increased access to patients (89%) and increased communication with families (62%). Following their training and actual facilitation of TBH, the 2 most frequently cited concerns with providing TBH were technical aspects of providing services (62%) and learning how to adapt their therapeutic style (57%).

Survey Comparisons

Three questions on the pretraining and post-implementation survey were further analyzed for significant differences in overall ratings. Ratings for motivation for providing TBH were significantly improved from the mean pretraining (M = 3.97, SD = 0.96) to post-implementation (M = 4.25, SD = 0.81); t(51) = 2.48, P = .02. A large effect size for this analysis was found (d = 0.81). Ratings for comfort screening patients to determine appropriateness of TBH were significantly improved from mean pretraining (M = 3.39, SD = 1.24) to post-implementation (M = 3.72, SD = 0.84); t(52) = 2.83, P = .01, d = 0.84. Ratings for confidence in facilitating TBH from pretraining to post-implementation were significantly different, with providers reporting more confidence post-implementation (pre-training M = 3.13, SD = 1.07; post-implementation M = 3.83, SD = 0.81); t(51) = 6.21, P = .00, d = 0.81.

Discussion

Due to restrictions related to the COVID-19 pandemic, face-to-face mental and behavioral health services in a large Midwestern children’s hospital were no longer able to be provided. In preparing providers for the transition to TBH, a series of training materials was developed. The present study sought to examine the effectiveness of this training initiative to rapidly train mental health providers at a large children’s hospital to provide TBH in response to the COVID-19 pandemic. Most providers were experienced in their field but had minimal previous experience in TBH.

Prior to the COVID-19 pandemic, this hospital had provided zero reimbursable TBH sessions within the past year. Following the commencement of the COVID-19 pandemic and move to TBH, the majority of providers reported completing at least 21 visits within the first month with a total of 1432 visits within 65 days. Following the completion of TBH training, providers endorsed significantly enhanced ratings regarding confidence in facilitating sessions, comfort screening patients, and motivation for providing services through TBH.

Despite the training initiative meeting its goals of helping providers become prepared to provide TBH in a rapid manner, providers still expressed concerns related to providing mental health care with this delivery model. The results of this study suggest that although the training materials were beneficial in preparing providers for initial implementation, providers remained interested in continued training opportunities.

Limitations

Participation in TBH training was voluntary and not all providers who accessed the training materials completed the pretraining or post-implementation surveys. Thus, results may not be representative of all providers who engaged in TBH service delivery. Given that the post-implementation survey was completed after both training and the start of providing services, results regarding comfort and confidence with the treatment modality should not be perceived as due to training alone. It is possible that the experience of providing services over TBH improved comfort and confidence as well.