Rapid, Full-Scale Change to Virtual PCIT During the COVID-19 Pandemic: Implementation and Clinical Implications.

Health agencies call for the immediate mobilization of existing interventions in response to numerous child and family mental health concerns that have arisen as result of the COVID-19 pandemic. Answering this call, this pilot study describes the rapid, full-scale change from a primarily clinic-based Parent-Child Interaction Therapy (PCIT) model to a virtual service model (i.e., I-PCIT) in an academic and community-based program in Miami, Florida. First, we describe the virtual service training model our program developed and its implementation with 17 therapists (MAge = 32.35, 88.2% female, 47.1% Hispanic) to enable our clinic to shift from providing virtual services to a small portion of the families served (29.1%) to all of the families served. Second, we examine the effect of I-PCIT on child and caregiver outcomes during the 2-month stay-at-home period between March 16, 2020, and May 16, 2020, in 86 families (MChildAge = 4.75, 71% Hispanic). Due to the rapid nature of the current study, all active participants were transferred to virtual services, and therefore there was no comparison or control group, and outcomes represent the most recently available scores and not treatment completion. Results reveal that I-PCIT reduced child externalizing and internalizing problems and caregiver stress, and increased parenting skills and child compliance with medium to large effects even in the midst of the COVID-19 pandemic. Finally, the study examined components of our virtual service training model associated with the greatest improvements in child and caregiver outcomes. Preliminary findings revealed that locally and collaboratively developed strategies (e.g., online communities of practice, training videos and guides) had the strongest association with child and caregiver outcomes. Implications for virtual service delivery, implementation, and practice in the midst of the COVID-19 pandemic are discussed.

Introduction

The 2019 coronavirus disease (COVID-19) is a major public health crisis that has resulted in increased fear, stress, risk for child maltreatment, and potential long-term mental health implications for children and families (Golberstein et al. 2020). During the COVID-19 pandemic, children and families have encountered significant life disruptions, including widespread school closures, remote learning transitions, and job losses that contribute to concerns about child and family mental health (Golberstein et al. 2020).

Children exposed to traumatic and/or stressful events, such as COVID-19, are at greater risk for experiencing emotional and behavioral problems (Rubens et al. 2018). Elevated levels of caregiver distress following stressful events also have been shown to negatively impact parenting practices and the caregiver–child relationship (Kelley et al. 2010). Families with children with preexisting emotional and behavioral problems and/or caregivers with preexisting psychological concerns may be particularly vulnerable to COVID-19-related stressors. In fact, recent work has highlighted the negative impact of the COVID-19 pandemic on level of stress and mental health of caregivers of children with chronic conditions (Tilburg et al. 2020). This is especially concerning for immigrant and minority communities that have been shown to be disproportionately impacted by COVID-19 (Clark et al. 2020). Therefore, interventions that foster positive caregiver–child interactions and reduce child behavioral problems and caregiver distress are needed in response to COVID-19, particularly for populations at greatest risk for negative outcomes. Parenting interventions represent one such promising set of interventions.

Disseminating Evidence-Based Treatments During COVID-19

Evidence-based face-to-face parenting interventions have historically been considered the gold standard approach for improving the caregiver–child relationship and child disruptive behaviors (Kaminski and Klaussen, 2017; Pantin et al. 2009; Piquero et al. 2016; Thomas et al. 2017). Given the effectiveness of these parenting interventions, efforts have been made to disseminate these interventions and evaluate what strategies are associated with successful implementation. However, face-to-face formats of service delivery often limit accessibility and availability of services (Danko et al. 2016; Nock and Ferriter, 2005; Owens et al. 2002). There have been multiple calls to diversify the service delivery formats due to the significant gap between the children who need services and the children who actually receive and complete services, especially among racial and ethnic minorities (Kazdin and Blase, 2011; McGoron and Ondersma, 2015; Ramos and Chavira, 2019). In an effort to increase accessibility (e.g., broader geographic reach, services occur in family’s natural environment) and reduce treatment barriers (e.g., lack of transportation, childcare, or schedule availability; McGoron and Ondersma, 2015), synchronous virtual delivery formats of parenting interventions have been developed.

Emerging research supports the effectiveness of virtual delivery of parenting interventions in improving treatment engagement, child disruptive behavior, and caregiver–child relationships (e.g., Dadds et al. 2019; Comer et al. 2017; Estrada et al. 2019; Prado et al. 2019). As it relates to virtual delivery of services for younger children, Parent–Child Interaction Therapy (PCIT), one of the most supported face-to-face parenting interventions (Kaminski and Claussen, 2017), has been examined via videoconferencing format (i.e., Internet-delivered PCIT [I-PCIT]) and found to be associated with improvements in child behavior and parenting skills in small open trials (Fleming et al. 2021; Kolhoff et al. 2020), and to have comparable treatment outcomes to face-to-face PCIT in a randomized controlled trial (RCT; Comer et al. 2017). While the emerging evidence for I-PCIT supports that this intervention is appropriate for children and families during COVID-19 (Gurwitch et al. 2020), little is known about what elements of virtual service delivery training promote therapist proficiency in this delivery format. In particular, evaluating principles of training and virtual service delivery will enhance understanding of how to improve the ecological validity of virtual parenting interventions for children during a pandemic (Comer and Myers, 2016).

Within PCIT’s emerging dissemination and training literature, within-agency trainers have been identified as key personnel for growing and sustaining delivery of PCIT services within organizations (Barnett et al. in press; Brabson et al. 2020). Didactics, behavioral rehearsal activities, fidelity checks, consultation, and supervision, when indicated, are endorsed as strategies to incorporate into therapist training (Beveridge et al. 2015; Brabson et al. 2020). Utilization of virtual synchronous video conference consultation from an expert trainer has been associated with modest improvements in child disruptive behavior above and beyond the effects of face-to-face PCIT services (Funderburk et al. 2015). Although there is emergent literature on strategies used to train therapists in face-to-face PCIT, there remains a need to examine what training strategies are appropriate for virtual service delivery and understand how these training strategies relate to family-level outcomes (Beveridge et al. 2015; Brabson et al. 2020).

Implementation Strategies

More generally, research on how to train therapists to deliver virtual services effectively, compared with other aspects of virtual service delivery, such as feasibility and cost effectiveness remains underdeveloped (Edirippulige and Armfield, 2017). Existing research describes discipline-specific training models delivered over extended periods of time (i.e., months) and often in context of an academic program (Felker et al. 2020; Traube et al. 2020). Considering the COVID-19 global pandemic, researchers have called for an acceleration in workforce capacity-building related to virtual services (Torous et al. 2020). Existing programs needed to be able to quickly and effectively deliver services virtually to continue to serve patients. To do so, training and capacity-building strategies for therapists should be effective in building competencies, amenable to therapists of different disciplines, and resource efficient. The Coalition for Technology in Behavioral Science (CTiBS) evidence-based framework for telebehavioral health competencies describes competencies that can be leveraged to advance workforce capacity-building in virtual service delivery (Maheu et al. 2017). This CTiBS framework guides supervisors and practitioners to develop training in virtual service delivery. Because the framework is interprofessional, measurable, and comprehensive, it may serve as a guide for how to support programs or therapists to become competent in delivering services virtually.

A review of studies on training and education to support virtual clinical services found that online learning was the most common approach and that there were few models which included hands-on practice opportunities (Edirippulige and Armfield, 2017). Yet, research consistently shows that didactic training alone is not sufficient to support effective implementation (Edmunds et al. 2013) Ongoing and additional supports, such as coaching and consultation, enhance the likelihood that an innovation will be implemented effectively (Edmunds et al. 2013). A recent study by Traube and colleagues examined a comprehensive virtual service training model for therapists implementing the Parents as Teachers home visiting model (Traube et al.2020). Therapists received 120 h of synchronous and asynchronous training in virtual service delivery, as well as role plays. Therapists identified the comprehensive training and availability of supervisors as facilitators to implementation and emphasized the importance of hands-on practice opportunities (Traube et al. 2020). This training model was developed and delivered as a package, over an extended period. The model provides a foundation of how to train therapists in virtual service delivery, but it is important to examine which supports within multi-component training models are most effective and how to quickly train therapists.

The Current Study

In response to the evolving pandemic and to minimize disruptions to care while mitigating exposure, our team utilized multiple strategies to build the capacity of PCIT therapists to rapidly shift to implementation of virtual services on March 16, 2020, when the stay-at-home order was instituted in Miami, a diverse city disproportionately impacted by COVID-19. The term virtual service delivery, as used in the current study, refers to the provision of synchronous telemental healthcare remotely, including secure videoconferencing and/or phone calls (Chou et al. 2016; Jent et al. 2018). In the current study, we present preliminary data to describe the rapid, full-scale change from a primarily clinic-based PCIT model to a virtual delivery model (i.e., I-PCIT), in response to the COVID-19 pandemic. Specifically, our program shifted from providing I-PCIT services to a small portion of our target population (with 17% of therapists providing services to 29.1% of program families) to providing I-PCIT services to all families.

First, we describe the virtual service training model and accompanying strategies our program developed, which were guided by the CTiBS framework, to implement this full-scale change to a virtual delivery model. Second, we examined the preliminary effect of I-PCIT on child and caregiver outcomes during the period between March 16, 2020, and May 16, 2020 (when the stay-at-home order was first lifted). Specifically, we expand on previous research examining the impact of I-PCIT on child behavior and caregiver burden (Comer et al. 2017) by examining outcomes in the following domains: (a) child externalizing and internalizing behavior and compliance; (b) parenting skills; and (c) caregiver stress. Finally, we examined what training strategies were associated with greatest improvements in these outcomes. We analyzed the training strategies in an exploratory fashion. Given the encouraging research findings supporting the use of consultation as a strategy to effectively train and support therapists delivering PCIT, and research that suggests practice opportunities are critical to effective training in virtual service delivery, we expected that therapist use of consultation would be associated with the greatest improvements in child and caregiver outcomes (Edirippulige and Armfield, 2017; Funderburk et al. 2015; Jackson et al. 2018).

Importantly, we use the term “outcomes” to describe child and caregiver behaviors at the end of the initial 2-month stay-at-home order. However, as noted below, these outcomes were measured after most families had received, on average, about half of all treatment sessions. We decided to collect and report on data over the course of this 2-month period to understand exactly how training effects took hold, and family outcomes changed, over the course of the pandemic stay-at-home order period. Results presented here are preliminary and should be interpreted with caution. Future studies are planned (and data are being collected currently) to follow-up with this sample after their full courses of treatment are completed and several months thereafter. We offer this caveat at the beginning of this manuscript in the interest of full transparency.

Methods

Setting and Participants

The academic and community-based program consists of six PCIT clinics, four that are embedded within the community and two that are embedded within the university hospital complex. The program was established in 2011 and is grant-funded to provide PCIT services free of charge to over 300 children aged 2 to 8 years and their families annually. Prior to March 2020, our program offered a hybrid approach to behavioral parenting interventions. Families self-selected into either (1) in-person, clinic-based, or (2) virtual services (i.e., I-PCIT).

The program’s six clinics consist of 17 master’s- and doctoral-level therapists with backgrounds in clinical psychology and mental health counseling. The program includes staff therapists (76.5%) and trainees (23.5%) who complete PCIT as an elective training rotation. Either a staff therapist or a trainee was the therapist of record, and analyses reflect the therapist of record. Therapists received PCIT training and weekly supervision with a Certified PCIT Trainer. Therapists had a mean age of 32.35 years (SD = 6.76 years; 88.2% female; 47.1% Hispanic, 88.2% White). All therapists had a master’s (64.7%) or doctorate (35.3%) degree, and 64.70% were Certified PCIT Therapists (i.e., PCIT International standards). Four therapists (23.5%) delivered virtual services prior to COVID-19. Patient participants included 86 children aged 2 to 8 years and their primary caregivers who were actively receiving services between March 16, 2020, and May 16, 2020. Demographic information is presented in Table 1.

Table 1

Dependent variable and covariate descriptive statistics and pre-COVID-19/during-COVID-19 comparisons

Dependent variables: treatment outcomes and parent skills
	Pre-COVID-19	During-COVID-19
	M (SD)	M (SD)	Hedges’ g
ECBI intensity score	146.57** (30.14)	111.79** (35.37)	1.05
BASC-3 internalizing T score	58.60** (11.91)	53.95** (11.19)	0.40
PSI-4 parent stress percentile	68.65** (18.33)	51.78** (27.09)	0.72
Do skills (total number)	4.79** (5.13)	23.38** (12.47)	− 1.93
Don’t skills (total number)	31.68** (18.39)	7.69** (7.44)	1.69
Effective command rate (%)	18.76** (15.67)	68.08** (25.86)	− 2.25
Follow-through rate (%)	0.77** (3.63)	64.18** (34.39)	− 2.56
Compliance rate (%)	47.81** (42.34)	85.48** (22.18)	− 1.09
Covariates
Child gender (% male)	74.42	N/A
Child age	4.75	1.62
Child race	79% White; 10.47% multiracial; 5.81% Black; 3.49% other, 1.16% native American
Child ethnicity	70.93% Hispanic/Latinx; 29.07% Non-Hispanic/Latinx
Parent gender (% male)	17.44	N/A
Parent race	81.40% White; 9.30% multiracial; 5.81% Black; 2.33% other, 1.16% Native American
Parent ethnicity	70.93% Hispanic/Latinx; 29.07% Non-Hispanic/Latinx
Parent education	6.98% HS diploma; 12.79% some college; 15.12% associates degree; 22.09% bachelor’s degree; 43.02% advanced degree
No. of treatment weeks	13.07	5.95
No. of treatment sessions	11.80	4.62
Pre-COVID-19 sessions	5.21	5.22
During-COVID-19 sessions	6.59	2.41

**p < .01 in paired-sample t test comparing pre- and during-COVID-19 scores. Hedges’ g, also known as the corrected effect size, is an effect size measure wherein values of < 0.2 indicate a small effect, values of approximately 0.5 indicate a medium effect and values > 0.8 indicate a large effect. In covariates section mean is listed in first column and standard deviation in second column unless %s are reported, then %s are reported in only 1 column

Procedure

Screening

Families were screened for services through a phone eligibility screening, which included providing verbal consent and completing a background form and behavioral questionnaires in English or Spanish. Inclusion criteria included (a) a child 2–8 years old, (b) primary caregiver fluent in English or Spanish, (c) elevated child disruptive behavior on the Eyberg Child Behavior Inventory Intensity Scale (ECBI; raw score ≥ 131; Eyberg and Pincus, 1999), or the Externalizing Problems subscales or composite of the Behavior Assessment System for Children, Third Edition (BASC-3 T score ≥ 60; Reynolds and Kamphaus, 2015).

Assessments

If inclusion criteria were met, families completed the intake assessment, which included written consent, completion of a clinical interview, and completion of questionnaires about their child’s functioning via REDCap. Families also participated in a 20-min behavioral observation to assess parenting skills and child compliance.

PCIT Intervention

Following the intake process, families began PCIT, which consisted of weekly assessment of child behavior and parenting skills taught to caregivers during two phases: Child-Directed Interaction (CDI) and Parent-Directed Interaction (PDI), over the course of 18 weeks. Therapists coach caregivers on their parenting skills from behind a one-way mirror via a wireless headset (for in-person services) or via videoconferencing (for virtual services).

Transition to Full-Scale Virtual Model

Implementation of I-PCIT prior to COVID-19

The academic and community-based PCIT program initially established I-PCIT as a subspecialty practice following brief consultation with Comer and colleagues (Comer et al. 2017). Given the promising findings of their RCT, we established a similar service delivery model with one exception (our PCIT program utilized internet-based audio for visits, not separate phone calls).

Developing a Virtual Service Training Model: Strategic Use of Workgroups

On March 13, 2020, our program decided to transition all in-clinic services to virtual starting March 16, 2020. To expedite the transition, program leadership established seven workgroups of two-to-four therapists with the goal of developing a virtual service training model. Workgroups were led by program champions, including therapists with prior experience delivering services virtually. Each workgroup was assigned a topic (e.g., developing a consultation framework for supporting therapists in virtual service delivery) and conducted extensive literature reviews, discussed the development and/or refinement of training protocols and administrative and clinical procedures, assessed therapist needs, and identified potential challenges to implementation. The workgroups used the competency domains of the CTiBS framework for telebehavioral health competencies to guide the development of the strategies and accompanying resources. For example, the Virtual Environment and Telepresence competency domain informed the development of the Skills Practice support (Support 4), in which the therapist completed practice scenarios to help support a family in setting up an optimal virtual environment. The workgroups identified eleven strategies (described in detail in Table 2) within 72 h of the transition to virtual services. The workgroups met during a weekly team meeting to discuss each workgroup’s recommendation and identify a plan for implementing the training strategies. On the same day, the full team was notified about the training strategies available to support them in the transition to virtual service delivery. Weekly workgroup meetings supported continued refinement of training strategies based on ongoing feedback.

Table 2

Describing and characterizing virtual implementation strategies

Independent variables: virtual implementation strategies
Name of support	% of families whose therapist participated	Description	Created internally or externally?
Strategy 1: web conference trainings from outside agencies	43.02%	Webinars and recorded trainings released by psychologists in the American Psychological Association (APA), Division 53, PCIT International, etc. Content varied from trainings specific to PCIT, to more general discussions of how to conduct virtual services successfully, including how to establish rapport and maintain privacy virtually	External
Strategy 2: recorded trainings developed by PCIT team	56.98%	The team shared recorded trainings created previously internally for conferences and training new therapists in I-PCIT	Internal
Strategy 3: one-on-one consultation	73.26%	Four therapists on the team who had previously conducted I-PCIT created a schedule of available “office hours” (approximately 10 h per therapist) each week for on-call consultation. Consultants helped therapists troubleshoot with families about both clinical and technological difficulties until therapists felt comfortable leading the troubleshooting on their own. After 1 month, this was discontinued, as therapists expressed confidence working through pitfalls on their own	Internal
Strategy 4: skills practice	41.86%	Therapists were given the opportunity to practice specific scenarios in a role-play with another therapist before needing to coach a client through the same scenario. Scenarios included unique difficulties that would occur in a virtual setting, including the parent having difficulty hearing the clinician, the call dropping unexpectedly, the child leaving the room, etc.	Internal
Strategy 5: shadowing cases	1.16%	Therapists new to virtual services shadowed the cases of experienced clinicians to observe the strategies they used to successfully complete PCIT virtually	Internal
Strategy 6: reviewing cases	26.74%	Videos of previous cases who received services virtually were available for therapists to review	Internal
Strategy 7: FAQ document	70.93%	As therapists reported the technological difficulties they encountered, consultants (the three clinic therapists with more than 5 h of prior training in virtual service delivery) recorded these problems and the corresponding solutions on a Google document accessible to the rest of the team	Internal
Strategy 8: online community of practice	80.23%	The clinic’s therapists met as a group to discuss common challenges encountered during I-PCIT, as well as ways to increase the strength of virtual PCIT. This group met weekly at the beginning of the stay-at-home order, and then biweekly. All trainees were encouraged to participate, both in the reporting of difficult therapeutic scenarios and in the generation of potential strategies for addressing the situations	Internal
Strategy 9: live observation and feedback	37.20%	For particularly difficult cases, or challenging sessions, therapists could request that a supervisor or I-PCIT consultant shadow them, joining them for the session	Internal
Strategy 10: virtual training materials (I-PCIT Guide)	69.77%	I-PCIT-experienced therapists on this team compiled and distributed a 53-page manual for transitioning PCIT successfully to virtual services	Internal
Strategy 11: in-session co-therapist support	31.40%	This clinic utilizes a co-therapy model to train new clinicians in PCIT. During the transition to I-PCIT, clinicians used this co-therapy structure to scaffold the training of new clinicians to become comfortable with I-PCIT as well	Internal

Virtual Service Training Model: Development and Implementation

Staff therapists were required to, at minimum, view locally and collaboratively developed recorded trainings on virtual service delivery (strategy 2), review a Frequently Asked Questions document during an existing team meeting (strategy 7), and attend an Online Community Practice (CoP) held during an existing team meeting (strategy 8). PCIT trainees completing an elective PCIT rotation were encouraged to utilize as many of the strategies available that would be helpful for their training but not required to complete minimum requirements in order to maintain and respect professional limits to their time allocated to the rotation. However, PCIT trainees were co-therapists with staff therapists who were exposed to at least minimum requirements. It was expected that staff therapists would model skills and share knowledge during co-therapy.

Data on therapists’ use of the 11 training strategies were collected using a self-reported online survey collected via REDCap on the week that the first stay-at-home order in Miami was lifted. The survey asked therapists to indicate (a) which of the 11 training strategies they used during the 2-month period wherein the stay-at-home order was in effect (from March 16, 2020, to May 16, 2020) and (b) the total estimated time they spent using the 11 strategies.

Putting Training into Practice

Notifying Participants

At the time data were abstracted, 85 families were actively receiving PCIT services. Out of the 85 families, 87.1% (n = 74) began receiving services pre-COVID-19. On March 16, therapists contacted the 74 families, 60 (81.1%) of whom were receiving in-clinic PCIT and 14 (18.9%) of whom were receiving I-PCIT pre-COVID-19, to inform them of the transition to virtual services and obtain information regarding their preference for continuing care. Families selected from the following options: (a) transition to I-PCIT or (b) transition to biweekly phone support. Out of the 60 families who were actively receiving in-clinic PCIT, 83.3% (n = 50) selected to transition to I-PCIT and 11.7% (n = 7) selected to transition to biweekly phone support. By May 2020, all but one family who initially selected biweekly phone support transitioned to I-PCIT. Families who were receiving I-PCIT pre-COVID-19 were given the option to continue I-PCIT or transition to biweekly phone support. All 14 families that were actively receiving I-PCIT pre-COVID-19 continued with I-PCIT.

Transitioning Participants to Virtual

For families who were actively receiving I-PCIT services pre-COVID-19, PCIT services remained relatively unchanged, except for a brief weekly check-in related to COVID-19 stressors. However, families who transitioned to I-PCIT from in-clinic services were required to complete a consent form for virtual services and a technology set-up session (i.e., test equipment and setting up home environment) prior to starting I-PCIT.

I-PCIT Session Structure

Caregivers completed all I-PCIT sessions online through a HIPAA-compliant Zoom session. Families used a computer with a webcam, tablet, or smartphone (angled to capture the play area) to broadcast live to the therapist, who provided live feedback and coaching on the caregiver’s skills during play-based interactions via headphones.

Measures

Eyberg Child Behavior Inventory (ECBI; Eyberg and Pincus, 1999)

The ECBI is a 36-item caregiver-report measure of disruptive behavior in children 2–16 years. The ECBI has demonstrated acceptable levels of reliability and validity in racially/ethnically diverse populations (α = 0.94, test–retest = 0.75; Gross et al. 2007) as well as stability over time and sensitivity to treatment change. Raw scores ≥ 131 (T score ≥ 60) on the Intensity Scale are considered clinically significant. The baseline ECBI Intensity Scale raw score was used to establish program eligibility and the most recent score was used as an outcome measure.

Behavior Assessment System for Children, Third Edition, Caregiver Rating Scale (BASC-3 PRS; Reynolds and Kamphaus, 2015)

The BASC-3 PRS is a caregiver-report measure of emotional, behavioral, and adaptive functioning of children aged 2–21 years. BASC-3 has well-established validity and reliability (α = 0.83–0.96, test–retest = 0.87–0.92; Reynolds and Kamphaus, 2015). T scores ≥ 70 are clinically significant and T scores 60–69 are considered at-risk. The Internalizing Problems composite score was used as an outcome measure.

Dyadic Parent–Child Interaction Coding System, Fourth Edition (DPICS-IV; Eyberg et al. 2013)

The DPICS-IV is a behavioral observation coding system that measures the quality of caregiver–child social interaction during 5-min play situations. Numerous studies have documented the reliability and validity of DPICS coding categories (Eyberg et al. 2013). To examine changes in parenting skills, caregiver verbalizations were coded for the frequency of positive statements (i.e., “Do Skills”: labeled praises, behavior descriptions, and reflections), and negative statements (i.e., “Don’t Skills”: questions, commands, and criticisms). Caregiver-led situations were coded for the rate of caregiver effective commands, the rate of caregiver correct follow-through on effective commands (i.e., praises for child compliance or time out warnings following noncompliance), and the rate of child compliance to effective commands. Coding was conducted live by PCIT therapists who were trained to 80% coding reliability. The DPICS-IV was used as an outcome measure of caregiver skill acquisition.

Parenting Stress Index, Fourth Edition: Short Form (PSI-4: SF; Abidin,2012)

The PSI-4: SF is a well-validated 36-item caregiver-report measure of parenting stress that has good internal consistency (α = 0.96; Abidin, 2012). The caregiver percentile score on the PSI-4: SF Total Parenting Stress Scale was used as an outcome measure of caregiver stress.

Analytic Plan

Paired-sample t tests were used to investigate whether PCIT improved child and caregiver outcomes between March 16, 2020, and May 16, 2020. These t tests identify whether there was a significant difference between baseline scores (“pre-COVID-19” scores) and most recently available scores prior to the May 16, 2020 (“during-COVID-19” scores; M = 13.19 weeks since baseline) in the following domains: (a) child behavior (i.e., ECBI Intensity Scale raw score, DPICS-IV Child Compliance Rate, BASC-3 Internalizing Subscale); (b) parenting skills (i.e., DPICS-IV Do and Don’t Skills, Effective Command Rate, Correct Follow-Through Rate); and (c) caregiver stress (i.e., PSI-4: SF Total Stress Scale).

We next conducted a series of iterative path analyses in MPlus Version 8.3 to understand what training strategies were associated with treatment outcomes and caregiver skill acquisition (Muthén and Muthén, 2017). These analyses proceeded in three steps. First, study covariates (i.e., child gender, race, ethnicity, and age, caregiver gender, ethnicity, race, and education, total number of treatment weeks and treatment sessions, and number of pre-COVID-19 treatment sessions) were entered into models predicting each during-COVID-19 dependent variable. To preserve study power and parsimony, only significant covariate associations at p < .05 were retained in further analyses. Second, pre-COVID-19 measures of the dependent variable were entered into the model. Third, variables representing therapist-reported use of each of the 11 training strategies were entered into each model predicting each of the 8 during-COVID-19 dependent variables. These were the final 8 models reported in “Results” (Table 3).

Table 3

Predicting during-COVID-19 outcomes from participation in specific virtual trainings controlling for pre-COVID-19 levels of outcomes

	During-COVID-19 outcomes
	ECBIβ (SE)	BASC-3 internalizingβ (SE)	PSI-4 parent stressβ (SE)	Do skillsβ (SE)	Don’t skillsβ (SE)	Effective command rateβ (SE)	Correct follow-through rateβ (SE)	Child compliance rateβ (SE)
Pre-COVID-19 level of outcome	0.44 (.07)*	0.38 (.11)*	0.44 (.12)*	0.25 (.07)*	0.19 (.19)	− 0.28 (.10)*	− 0.17 (.09)	− 0.03 (.16)
Strategy 1: web conference trainings from outside agencies	0.28 (.10)*	0.95 (.22)*	0.45 (.14)*	0.03 (.15)	0.38 (.15)*	− 0.10 (.15)	− 0.07 (.22)	0.84 (.14)*
Strategy 2: recorded trainings developed by PCIT team	− 0.61 (.10)*	− 0.78 (.11)*	− 0.30 (.10)*	0.34 (.08)*	− 0.37 (.12)*	0.02 (.11)	− 0.29 (.24)	− 0.04 (.08)
Strategy 3: one-on-one consultation	0.18 (.12)	0.17 (.16)	− 0.12 (.19)	− 0.30 (.08)*	0.33 (.14)*	− 0.10 (.11)	− 0.15 (.10)	0.12 (.07)
Strategy 4: skills practice	0.44 (.15)*	− 0.28 (.21)	− 0.31 (.27)	− 0.54 (.15)*	− 0.53 (.18)*	− 0.17 (.11)	− 0.16 (.18)	− 0.25 (.14)
Strategy 5: shadowing cases	− 0.20 (.11)	− 0.15 (.09)	N/A	0.28 (.15)	− 0.03 (.07)	0.17 (.10)	− 0.06 (.04)	N/A
Strategy 6: reviewing cases	0.19 (.10)*	0.90 (.19)*	0.17 (.22)	0.13 (.08)	0.33 (.15)*	0.06 (.07)	0.38 (.24)	0.18 (.05)*
Strategy 7: FAQ document	0.29 (.10)*	0.85 (.28)*	0.59 (.16)*	− 0.19 (.08)*	− 0.14 (.10)	− 0.06 (.08)	− 0.56 (.11)*	− 0.11 (.14)
Strategy 8: online community of practice	− 0.26 (.12)*	− 0.77 (.32)*	− 0.60 (.23)*	0.11 (.11)	− 0.07 (.15)	− 0.06 (.08)	0.24 (.09)*	0.04 (.07)
Strategy 9: live observation and feedback	− 0.06 (.07)	− 0.21 (.15)	0.05 (.11)	0.00 (.07)	0.27 (.10)*	− 0.10 (.11)	− 0.23 (.10)*	− 0.17 (.17)
Strategy 10: virtual training materials (I-PCIT guide)	− 0.35 (.16)*	0.15 (.18)	0.04 (.21)	0.43 (.14)*	0.20 (.14)	− 0.05 (.08)	0.47 (.13)*	0.33 (.10)*
Strategy 11: in-session co-therapist support	− 0.08 (.07)	0.01 (.17)	0.07 (.20)	0.17 (.10)	0.08 (.08)	0.06 (.12)	0.31 (.10)*	− 0.11 (.15)
Child ethnicity	N/A	N/A	N/A	N/A	N/A	− 0.62 (.14)*	N/A	N/A
Child age	N/A	N/A	N/A	N/A	N/A	− 0.54 (.15)*	N/A	0.23 (.09)*
Parent gender	− 0.01 (.07)	− 0.16 (.10)	N/A	N/A	N/A	− 0.25 (.10)*	N/A	− 0.31 (.07)*
Parent education	N/A	N/A	N/A	0.26 (.11)*	N/A	N/A	N/A	N/A
Parent ethnicity	N/A	N/A	N/A	N/A	− 0.12 (.07)	0.54 (.14)*	N/A	N/A
No. of treatment weeks	N/A	N/A	N/A	N/A	N/A	− 0.62 (.24)*	N/A	− 0.60 (.24)*
No. of treatment sessions	N/A	N/A	N/A	N/A	− 0.41 (.09)*	0.54 (.21)*	N/A	0.41 (.19)*
No. of pre-COVID-19 sessions	N/A	− 0.25 (.09)*	N/A	N/A	N/A	N/A	N/A	N/A

*p < .05, N/A indicates parameter estimate was not included in the final model because it did not significantly (p < .05) predict the outcome in an initial model only including covariates (i.e., child ethnicity, age, parent gender, ethnicity, education, number of treatment weeks, number of treatment sessions, number of pre-COVID-19 treatment sessions). Covariates not significant in this initial model were trimmed from final model in interest of model parsimony and to ensure final model was estimable. Support 5 is marked as N/A in two models where it was not estimable because participants whose therapists participated in that training had not completed that measure yet

In accordance with expert recommendations (Muthén and Muthén, 2017), full information maximum likelihood (FIML) estimation procedures were utilized to account for data missingness, and nesting of families within therapist was accounted for by adjusting standard errors of estimates utilizing the type = COMPLEX MPlus algorithm. Controlling for nesting of families within therapist controls for therapist-level effects on during-COVID-19 outcomes (i.e., differences in baseline therapist skill level, comfort with technology, experience with virtual service delivery, etc.). Controlling for prior number of weeks, treatment sessions, and pre-COVID-19 treatment sessions protects against the concern that significant differences in during-COVID-19 measures were due to a greater number of pre-COVID-19 sessions attended by families. Controlling for pre-COVID-19 measures of each dependent variable ensures that any virtual service training effects emerge above and beyond baseline levels of behavior, as recommended by intervention scientists (e.g., De Los Reyes, 2017).

Results

Participant missing data on any during-COVID-19-dependent variable did not significantly differ from participants with complete data on any covariate, baseline treatment outcome or caregiver skill variable, training strategy variable, or specific therapist. Thus, no problematic missingness emerged in these data. Descriptive data on all study variables are reported in Tables 1 and 2, and correlations between study variables are reported in Supplemental Table 1. Notably, when the stay-at-home order took effect, families had completed, on average, 5.21 total sessions. However, the first two to three sessions are intake sessions, and the third (or fourth) session is a “teach” session where caregivers didactically learn skills from the therapist. Therefore, on average families completed 2.65 actual treatment sessions (wherein caregiver, therapist, and child were all present and the caregiver was actively practicing the skills) before the pandemic stay-at-home order came down. Over the course of the stay-at-home order, families completed, on average, an additional 6.59 sessions (Table 1). Therefore, most families in this study had recently started treatment as the PCIT program as the stay-at-home order was put into place. Consequently, the child and adult “outcomes” presented here demonstrate approximately mid-treatment progress and should be considered preliminary and interpreted with caution.

Evaluating PCIT Efficacy

Paired-sample t tests revealed that during-COVID-19 measures demonstrated significant improvement compared with pre-COVID-19 measures on all child and caregiver outcomes (Table 1). Specifically, with regards to child outcomes, child ECBI scores improved by an average of 34.59 points (t(83) = 10.19, p < .01), child compliance rate increased by an average of 50.15 percentage points (t(32) = 6.13, p < .01), and child BASC-3 Internalizing Problems Composite t scores improved by an average of 4.36 points (t(53) = 4.42, p < .01).

With regard to caregiver outcomes, PSI-4 Total Stress percentile scores improved by an average of 18.16 points (t(59) = 18.17, p < .01). Caregiver skills also improved. Total caregiver positive statements (“Do Skills”) increased by an average of 18.75 skills (t(83) = 14.71, p <  .01), caregiver negative statements (“Don’t Skills”) decreased by an average of 23.43 skills (t(83) = 12.05, p < .01), caregiver effective commands rate increased by an average of 53.30 percentage points (t(33) = 9.49, p < .01), and caregiver correct follow-through rate increased by an average of 63.33 percentage points (t(53) = 10.54, p < .01). The effect sizes of all child and caregiver outcomes were medium-to-large (see Hedges’ g section of Table 1; Cohen, 1988).

Associations Between Training Strategies and PCIT Outcomes

Next, we tried to ascertain whether specific strategies of the virtual service training model were associated with observed improvements in PCIT outcomes over the course of the COVID-19 pandemic. All results are reported in Table 3 as standardized parameter estimates. All results control for pre-COVID-19 levels of each outcome and nesting of families within therapists. Larger patterns are reported in prose. For detailed results refer to Table 3. Notably, all therapists completed at least one strategy, 71.6% of therapists completed three or more strategies, and cumulatively, therapists reported that they completed 4.82 h of virtual service training.

Child Outcomes

During-COVID-19 ECBI Scores

Therapists viewing locally and collaboratively developed recorded trainings on virtual service delivery, participating in a CoP, and utilizing a locally developed I-PCIT guide were all associated with significantly decreased ECBI scores, even after controlling for pre-COVID-19 scores and caregiver gender (Table 3). However, therapists’ viewing web-based trainings developed by outside agencies, role-playing skills, reviewing videos of prior cases, and utilizing a FAQ document were all associated with significantly increased ECBI scores after controlling for participation in other training strategies.

During-COVID-19 Child Compliance

Therapists viewing web-based trainings developed by outside agencies, reviewing videos of prior cases, and using a locally developed I-PCIT guide were all associated with higher child compliance.

During-COVID-19 BASC-3 Internalizing Problems Composite Scores

Therapists viewing locally and collaboratively developed recorded trainings and participating in a CoP were each associated with significantly decreased BASC-3 Internalizing scores. However, therapist viewing web-based trainings developed by outside agencies, reviewing videos of prior cases, and using a FAQ document were associated with increased BASC-3 Internalizing scores (Table 3).

Caregiver Outcomes

During-COVID-19 PSI-4: SF Total Parenting Stress Scale Scores

Therapists viewing locally and collaboratively developed recorded trainings and participating in a CoP were both associated with significantly lower Total Parenting Stress scores, whereas therapists viewing web-based trainings from outside agencies and utilizing a FAQ document were both associated with significant increases in these scores (Table 3).

During-COVID-19 Positive “Do” Skills

Therapists viewing locally and collaboratively developed recorded trainings and utilizing a locally developed I-PCIT guide were associated with significant increases in caregiver Do Skills. However, consultations, role-plays, and use of a FAQ document were associated with significantly lower caregiver Do Skills (Table 3).

During-COVID-19 Negative “Don’t” Skills

Therapists viewing locally and collaboratively developed recorded trainings and role-playing their skills was related to significant decreases in Don’t Skills. However, therapists viewing web-based trainings from outside agencies, participating in consultations, reviewing videos of prior cases, and receiving live observation and feedback were associated with significantly higher Don’t Skills.

During-COVID-19 Effective Command Rate

No training strategy was significantly associated with during-COVID-19 Effective Command Rate.

During-COVID-19 Correct Follow-Through Rate

Therapist participation in a CoP, use of a locally developed I-PCIT guide, and in-session co-therapist support were associated with significantly higher correct follow-through rates. However, use of a FAQ document and live observation and feedback were associated with lower correct follow-through rates.

Summary Across All Outcomes

Specific training strategies appeared to be broadly beneficial across all during-COVID-19 outcomes. Therapists viewing locally and collaboratively developed recorded trainings was associated with improvements in all caregiver and child treatment outcomes, and 2 out of 5 parenting skills. Similarly, therapists participating in a locally coordinated CoP was associated with significant improvements in all treatment outcomes. Finally, therapist use of a locally developed I-PCIT guide was associated with improved ECBI scores, caregiver correct follow-through rate, and child compliance rate. Unfortunately, other training strategies were associated with negative treatment outcomes. Therapist use of web-based trainings developed outside of the local environment was associated with worsening of all three treatment outcomes. Therapist use of a FAQ document also was associated with increases in all three treatment outcomes as well as decreases in Do Skills and Correct Follow-Through rates.

Discussion

In an effort to answer calls for immediate service mobilization (Holmes et al. 2020), while aligning with the CTiBS framework (Maheu et al. 2017) and identified implementation strategies (Powell et al. 2014), we expanded the virtual service delivery capacity of our team to meet the mental health needs of children and their families in our high-risk community during the COVID-19 pandemic. While researchers have called for an acceleration in workforce capacity building related to delivering services virtually (Torous et al. 2020), little is known about what strategies are most effective at promoting successful rapid implementation and how these methods may impact patient outcomes. The current study is a first step in addressing this gap (e.g., Edmunds et al. 2013).

Findings revealed that children demonstrated large reduction in caregiver-reported disruptive behaviors (Table 1). This magnitude of change is similar to that observed by Comer et al. (2017) for children receiving both clinic-based and I-PCIT (i.e., large to very large effects), with the current study being the first to provide evidence for the effectiveness of I-PCIT for families who transitioned from in-clinic to virtual services and, importantly, who made this transition during the COVID-19 pandemic. Of note, the second time point for the current study occurred prior to the end of the PCIT protocol, with families on average being about halfway through treatment, and should therefore be interpreted with caution. Consistent with caregiver-reported changes in child externalizing behavior, children were also observed to be significantly more compliant, and this effect size was similarly large.

Additionally, significant decreases were found for internalizing behavior problems, with medium effect size. Findings are consistent with Chase and Eyberg (2008), with reductions in internalizing symptoms following in-clinic PCIT, even among children with baseline subclinical internalizing symptoms. The current study adds to the existing literature by demonstrating that even without adaptations specifically geared to addressing internalizing symptoms, I-PCIT can still positively impact internalizing symptoms. This is especially important in the context of COVID-19, as children often experience increases in both externalizing and internalizing symptoms during stressful and/or traumatic events (Rubens, Felix and Hambrick, 2018).

Additionally, caregivers reported significant and large reductions in parenting stress and showed improvements in their parenting practices. Specifically, caregivers were observed to make more positive statements and fewer negative statements toward their children at the second time point, and they were observed to issue more effective commands with appropriate follow through. These large effects highlight PCIT’s potential to impact both actors in the bidirectional caregiver-child relationship (Thomas et al. 2017). Not only does PCIT improve child behavior and parenting skills, but also our data demonstrate it also significantly reduces parenting stress during times of unprecedented uncertainty, such as the COVID-19 pandemic.

The aforementioned encouraging findings provide preliminary support for the use of PCIT in helping vulnerable families adapt to COVID-19-related challenges (e.g., sickness, working from home, school closures, routine disruption) that otherwise may have significant and deleterious effects on family mental health. Moreover, preliminary findings related to the efficacy of I-PCIT in the midst of a global pandemic fare well relative to other clinic-based PCIT studies of children with behavior problems (McNeil and Hembree-Kigin, 2010).

In terms of the association between the virtual service training model and outcomes, findings suggest locally and collaboratively developed strategies (i.e., recorded trainings on virtual service delivery, CoP, and I-PCIT guide), two of which were required of staff therapists, appeared to have the most consistent strong association with family-level outcomes. However, other training strategies appeared to have varying impacts on treatment outcomes. This was especially evident for training strategies that were developed outside of the local environment, which were at times found to have a negative association with family-level outcomes. These conflicting findings highlight the importance of examining the implementation processes in relation to the larger system in which they are embedded (Glisson and Williams 2015). Our results are consistent with research that suggests inner-setting factors (e.g., co-creating, collaborating) play a key role in training transfer (Jackson et al. 2019). These findings also are consistent with Baldwin and Ford’s training transfer model (1988), which hypothesizes that training transfer occurs as a result of three factors: training design (e.g., content of training), individual trainee characteristics (e.g., demographics, skills), and work environment (e.g., organizational climate, support, opportunities to utilize skills). In the current study, therapists played a critical role in the design and organization of certain training strategies, which also likely contributed to maintaining a supportive and collaborative organizational climate. This may in part explain why locally and collaboratively developed strategies were more likely to have contributed to training transfer and ultimately to positive effects on outcomes. Another possible explanation for the conflicting findings is that training strategies that were externally developed were less adapted to local context and less practical in nature. Taken together, results highlight the importance of co-creating and collaborating in the context of training and professional development.

Despite the positive effect associated with locally and collaboratively developed strategies, some locally and collaboratively developed strategies also were found to have varying associations with treatment outcomes. These strategies tended to focus on providing additional practice and resources (e.g., skill practice, case review, consultation, FAQ). It is possible that conflicting findings may be related to the process by which therapists engaged with these supports, as conclusions involving the temporal direction of training strategy utilization cannot be made. Above and beyond the required training strategies for staff therapists, both staff therapists and trainees self-selected the training strategies. It is possible that findings may be attributable to therapists utilizing training strategies in a tiered manner, where locally and collaboratively developed strategies served as a core component of initial training and other strategies related to ongoing support were utilized if additional support was needed.

Strengths and Limitations

It is important to interpret the findings in light of several limitations. First, the virtual service training model was developed and implemented within a university-based clinic setting where there was an existing established virtual service delivery structure, which likely contributed to the success in rapidly transitioning our program to full-scale virtual services. It is possible that findings may not generalize to community and other settings, where the same level of implementation support may not be feasible, and the extent of existing infrastructure for virtual services may vary. Additionally, the sample predominantly identified as Hispanic. Despite limitations with regard to generalizability to other ethnic groups, findings contribute to the existing literature as Hispanic children are often understudied in psychosocial intervention research (La Greca, Silverman, and Lochman, 2009). Therefore, it is important for future research to examine whether these findings extend to other settings and ethnic groups. Second, the sample size was relatively small given that only participants who were actively receiving services both in-clinic and virtually during the 2-month period between March 16, 2020, and May 16, 2020, were included. Therefore, the data presented should be interpreted as preliminary and in need of replication in larger samples. Third, the lack of a control group limits the ability to make inferences about changes in child and caregiver functioning. Changes in child and caregiver outcomes may be attributable to regression to the mean or a social desirability bias in responding rather than intervention effects. However, given research supporting higher levels of externalizing and internalizing behavior problems among children exposed to stress, trauma, and/or disaster (Rubens et al. 2018), it is unlikely that symptoms would have resolved naturally.

Fourth, data analyzed were from the primary caregiver and relied on caregiver report, particularly for child internalizing behavior. Thus, future studies should incorporate multimodal assessments that include multiple informants allowing for a more comprehensive evaluation of child behavior. Fifth, due to the rapid nature of the current study and timing at which data was analyzed, during-COVID-19 scores represent the most recently available scores approximately halfway through treatment for most families. Thus, it is unclear whether intervention effects will be maintained over time. Nonetheless, this study was an important first step in understanding the effectiveness of PCIT in improving child and caregiver outcomes during a global health crisis.

Another limitation was the rare therapist use of the “shadowing cases” training support (only 1.16% of families had therapists who used this support). Though we continued to include this training support in study analyses to ensure results are reported transparently, null findings concerning this training support should be interpreted extremely cautiously and future studies with larger samples are needed to investigate this support’s efficacy. A final limitation was that the current study was underpowered to examine potential mediating processes that might explain how therapist participation in training strategies led to better or worse family outcomes. Future studies could utilize the Integrated Model of Program Implementation (Berkel, Mauricio, Schoenfelder, and Sandler, 2011) to examine indicators of parent responsiveness (e.g., attendance rate, active participation, home practice, and satisfaction) as potential mediators.

Despite these limitations, the present study is the first to outline the implementation process required to rapidly transitioning PCIT to virtual service delivery and examine the strategies associated with greatest improvements in family-level outcomes. Moreover, our training approach is aligned with the CTiBS framework, allowing for replicability of our model. The current study is also among the first to link training strategy use with both reported and observed child and caregiver outcomes, allowing for examination of the translation of training to therapeutic effect. Finally, our study captured child and family functioning at pre-pandemic baseline levels, allowing us to assess the impact of our program in the midst of a pandemic.

Clinical Implications

The current study provides preliminary findings with important clinical implications for the rapid, effective mobilization of resources to provide widely accessible mental health services to children and families during a global pandemic. An overwhelming amount of research indicates that training alone is insufficient to support successful implementation (Edmunds et al. 2013). The current study adds to this work by outlining the training strategies found most effective in producing positive change in domains commonly affected by stressful and/or traumatic events. Given the varying associations with treatment outcomes for some training strategies, it is recommended that future training efforts promote a hierarchal approach to training, where universally beneficial training strategies that have been shown to have the strongest association with positive family-level outcomes (e.g., locally and collaboratively developed strategies) are used to build a foundation of knowledge related to virtual service delivery and situationally beneficial strategies (e.g., skill practice, case review, consultation, FAQ) are used when more specific supports are needed. Taken together, significant improvements in externalizing and internalizing child behavior as well as caregiver stress and parenting skills over time and in the context of a global pandemic build on the existing PCIT dissemination and training literature and highlight the potential for I-PCIT to effectively address the mental health needs of children and their families. Continued efforts are needed to leverage technology innovations to rapidly increase access to much-needed interventions such as PCIT, particularly in the context of a global pandemic.

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 16 KB)